System for Linking to Documents with Associated Annotations

ABSTRACT

An annotation system includes first annotation data for annotating a manifestation of a first instance of a first XML document. The first instance of the first XML document is served by a document system. The first annotation data includes a first XML document identifier for the first XML document. The first annotation data is uniquely identified by a first annotation data identifier. The document system is configured to respond to a request containing the first XML document identifier with a manifestation of a second instance of the first XML document. The annotation system: (A) receives a request containing the first annotation data identifier; (B) issues a request containing the first XML document identifier to the document system; and (C) in response to the request containing the first annotation data identifier, manifests the first annotation data in connection with a manifestation of a second instance of the XML document.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to the following commonly owned andco-pending patent applications, all filed concurrently herewith, andidentified below by their titles and attorney docket numbers:

-   -   “System for Annotating Documents Served by a Document System        without Functional Dependence on the Document System,” Attorney        Docket Number G0006-1001;    -   “System for Creating and Editing Temporal Annotations of        Documents,” Attorney Docket Number G0006-1002; and    -   “System for Programmatically Accessing Document Annotations,”        Attorney Docket Number G0006-1003.

BACKGROUND

Massive computer networks, ready access to such communications byconsumers and businesses alike, and the prevalence of personal computingdevices have made access to multimedia documents widespread. Computerssupporting graphical user interfaces and audio have made web pagesincorporating multi-media content a common form of document and webpages represent a common interface for computer programs. Giganticnumbers of online multimedia documents constitute a massive amount ofcontent accessible to hundreds of millions of users. Currently however,as is well understood by those skilled in the art, whilst the user mayview such online documents, and thereby passively consume such content,the user's ability to actively engage with such content is extremelylimited. There has been a trend for site publishers to encourage userengagement by such means as blogs, forums and bulletin boards and suchmeans have indeed resulting in a massive explosion of user generatedcontent but such attempts to go beyond the passive consumption ofpackaged content do not represent a powerful general model forinteraction with content because the user's ability to comment oncontent in such cases is limited, typically constrained to featuresprovided by the web site implementation and authorised by the websitepublisher. Such limitations dictate which content can be commented on atwhat time and in what way. The general definition of ‘annotation’ isextra information associated with a particular point in a document andhence we note that such limited user interaction with online contentconstitutes a weak form of annotation—the forms of extra information arelimited, the points that may be annotated are limited, and the documentsthat may be annotated are limited.

SUMMARY

In one embodiment, an annotation system includes first annotation datafor annotating a manifestation of a first instance of a first XMLdocument. The first instance of the first XML document is served by adocument system. The first annotation data includes a first XML documentidentifier for the first XML document. The first annotation data isuniquely identified by a first annotation data identifier. The documentsystem is configured to respond to a request containing the first XMLdocument identifier with a manifestation of a second instance of thefirst XML document. A method for use with the annotation systemincludes: (A) receiving a request containing the first annotation dataidentifier; (B) issuing a request to the document system, wherein therequest contains the first XML document identifier; and (C) in responseto the request containing the first annotation data identifier,manifesting the first annotation data in connection with a manifestationof a second instance of the XML document.

The first data may include first annotation data representing a firstassociation between: (a) a first manifestation of first annotationcontent in connection with a manifestation of the first instance of thefirst XML document and (b) a manifestation of a first instance of afirst target in the manifestation of the first instance of a first XMLdocument. The first UALI may include a first Unique AnnotationIdentifier (UAI) that uniquely identifies the first annotation data.Operation (A) may include receiving a request containing the first UAI,and operation (C) may include, in response to the request containing thefirst UAI, manifesting the first annotation data in connection with amanifestation of a second instance of the first XML document. The firstannotation data may include first annotation content data representingthe first annotation content, first target locating data for locatingthe second instance of the first target, and first association datarepresenting the first association.

The first data may include first annotation set data representing afirst annotation set. The first UALI may include a first Unique SetIdentifier (USI) that uniquely identifies the first annotation set.Operation (A) may include receiving a request containing the first USI,and operation (C) may include, in response to the request containing thefirst USI, manifesting the first annotation set data in connection witha manifestation of a second instance of the first XML document. Thefirst annotation set may further contain second annotation data forannotating the first instance of the first XML document; and operation(C) may further include, in response to the request containing the firstUSI, manifesting the second annotation data in connection with themanifestation of the second instance of the first XML document. Theannotation set data may further include second annotation data forannotating the first instance of the first XML document; and operation(A) may further include, in response to activation of the hyperlink,manifesting the second annotation data in connection with themanifestation of the second instance of the first XML document.

Operation (A) may include: (A)(1) at a client device: (A)(1)(a)receiving input containing the first UALI; (A)(1)(b) retrieving thefirst data from storage; (A)(1)(c) requesting the second instance of thefirst XML document from the document server; and (A)(1)(d) manifestingthe first data in connection with the manifestation of the secondinstance of the first XML document. Alternatively, operation (A) mayinclude: (A)(1) at a client device: (A)(1)(a) receiving input containingthe first UALI; (A)(1)(b) transmitting a request containing the firstUALI to an annotation server; (A)(2) at the annotation server (A)(2)(a)requesting the second instance of the first XML document from thedocument server; (A)(2)(b) creating a composite of the first data andthe second instance of the first XML document; (A)(2)(c) transmittingthe composite to the client device; and (A)(3) at the client device:(A)(3)(a) manifesting the composite. Alternatively, operation (A) mayinclude: (A)(1) at a client device: (A)(1)(a) receiving input containingthe first UALI; (A)(1)(b) transmitting a request containing the firstUALI to an annotation server; (A)(1)(c) requesting the second instanceof the first XML document from the document server; (A)(1)(d)transmitting the first data and the second instance of the first XMLdocument to the annotation server; (A)(2) at the annotation server:(A)(2)(a) creating a composite of the first data and the first XMLdocument; (A)(2)(b) transmitting the composite to the client device; and(A)(3) at the client device: (A)(3)(a) manifesting the composite.

The first annotation data may represent a first association between: (a)a first manifestation of first annotation content in connection with amanifestation of the first instance of the first XML document and (b) amanifestation of a first instance of a first target in the manifestationof the first instance of a first XML document. The first annotationcontent may include first manifestation data, and the firstmanifestation of the first annotation content may be a manifestation ofthe first manifestation data.

The first manifestation data may be or include at least one of text, animage, audio, video, a form element, a hyperlink, JavaScript, and anembedded object. The first annotation content may further include firstmanifestation control data include a specification of how the programmanifests the annotation content. The first manifestation control datamay include one or more of first manifestation timing data to specifythe temporal behavior of the manifestation of the first annotationcontent; first manifestation realization data to specify manifestationbehavior of the first annotation content; and first manifestationexternal interface specification data to specify conditions for accessto the annotation data and annotation behavior in response to externalevents.

The first instance of the first target may be or include one or more ofthe following: text, an image, audio, video, a form element, ahyperlink, JavaScript, an embedded object, a document, a documentobject, a region in a document object, a point in a document object, acollection of related document objects, and a region which spans acollection of related document objects. The first instance of the firstXML document may be a web page. The first instance of the first targetmay be the web page. The first instance of the first target may be orinclude one or more of the following: a page object, a region in a pageobject, a point in a page object, a collection of related page objects,and a region which spans a collection of related page objects.

The first annotation data may represent a first association between: (a)a first manifestation of first annotation content in connection with amanifestation of the first instance of the first XML document and (b) amanifestation of a first instance of a first target in the manifestationof the first instance of a first XML document, wherein the firstassociation comprises a spatial association between the firstmanifestation of the first annotation content and the manifestation ofthe first instance of the first target. The first association may be atemporal association and/or a spatial association between the firstmanifestation of the first annotation content and the manifestation ofthe first instance of the first target.

The first annotation data may be stored: in a distinct file from thefirst instance of the first XML document; on an annotation server thatis distinct from the document server; by the annotation system withoutmodifying the document server; by the annotation system withoutmodifying any software on the document server; by the annotation systemwithout modifying any content on the document server.

Other features and advantages of various aspects and embodiments of thepresent invention will become apparent from the following descriptionand from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary high level block diagram of a system enabling thestorage, retrieval, transmission, processing and display of documentsaccording to one embodiment of the present invention.

FIG. 2 is a high level block diagram of a document system and afunctionally distinct annotation system, wherein the annotation systemmay annotate documents managed by the document system according to oneembodiment of the present invention.

FIG. 3 illustrates the use of manifestation control data to influencethe timing of the manifestation of annotations according to oneembodiment of the present invention.

FIG. 4 shows how the position of manifested annotation content on thepage is determined by its spatial offset from the target according toone embodiment of the present invention.

FIG. 5 shows how the time when the annotation content is manifested onthe page is determined by its target on the annotated page objecttimeline and any temporal offset from this target according to oneembodiment of the present invention.

FIG. 6 is a diagrammatic representation of the annotation data structureaccording to one embodiment of the present invention.

FIG. 7 shows a multi-component annotation on web page according to oneembodiment of the present invention.

FIG. 8 illustrates the use of both the ‘temporal offset’ and ‘MCD delay’for controlling the temporal behavior of an annotation according to oneembodiment of the present invention.

FIG. 9 is a flowchart depicting a high level view of the program'srestore process according to one embodiment of the present invention.

FIG. 10 is a diagram depicting the annotation save process according toone embodiment of the present invention.

FIG. 11 is a high level overview of how an annotation set is restoredaccording to one embodiment of the present invention.

FIG. 12 is a diagram showing a simple example of how annotationprogramming and communication between cooperating annotations can beused to implement a dynamic manifestation according to one embodiment ofthe present invention.

FIG. 13 is a high level diagram of the system architecture ofcontemporary social plugins that may be used in connection with oneembodiment of the present invention.

FIG. 14 is a high level diagram of a novel system architecture for theincorporation of social data into third party websites by means of novelannotations implemented according to embodiments of the presentinvention.

FIG. 15 is a depiction of the effects of programmatically editingelements of the annotation triple according to one embodiment of thepresent invention.

FIG. 16 is a depiction of the relationship between a USI hyperlink, theannotation set data, and the underlying document data according to oneembodiment of the present invention.

FIG. 17 is an illustration of semantic integrity according to oneembodiment of the present invention.

FIG. 18 is a depiction of the relationship between a UAI hyperlink, theannotation data, and the underlying document data according to oneembodiment of the present invention.

DETAILED DESCRIPTION

A user should be able to annotate any web page content, of any mediatype, on any web page, on any website, at any time, with any number ofannotations of any media type; to be able to save, retrieve and sharesuch annotations, and to do all this without requiring the support, oreven the permission, of the site publisher, without taking copies of thecontent, and without changing the content, the site itself (includingany software being used to support the site) or the browser softwareused to view the content. As will be evident to those skilled in the artthis is a much stronger model of annotation for web page content thancurrently exists.

The strong functional independence of the annotation system from thewebsite and system that serves content to the website is an extremelyimportant characteristic of certain embodiments of the invention; itmeans that such embodiments of the invention are able to providevaluable functionality to users without demanding changes to existingweb infrastructure and may therefore readily made available.Furthermore, only with such independence may an annotation system enableusers to interact with web page content without requiring thecooperation and permission of the site publisher.

In the case of embodiments of the present invention a document to beannotated may, for example, be a web page and the type of extrainformation which may be used to annotate web pages should beinterpreted very broadly. Annotations may include not just textcommentary on original page content, but much richer annotation such asmulti-media augmentation of (or complete changes to) this content,linking the content to other web pages, and enabling additionalinteraction with content (e.g., by the introduction of new hyperlinks.).

Some systems exist that allow annotating web pages and savingannotations. For example, the most popular web page annotation modessupported by existing annotation systems have been (a) post-it stylenotes, (b) drawing, and (c) text highlighting. Post-it style notesenable the user to drop a notelet onto a web page, to edit that noteletand save and retrieve it. An example of such notelet technology may befound at www.mystickies.com. Drawing onto a web page is less common butmay be found at www.drawhere.com. Perhaps the most common annotationmode is text highlighting which may be found, for example, atwww.diigo.com or www.sharedcopy.com. Diigo further enables theassociation of notelets with highlighted text. The sitewww.shiftspace.com provides software to enable users to attach notes,highlight text, change photographs and edit underlying HTML.

There are also existing systems that recognise the utility of persistentannotations. Typically such annotations are saved to and restored from aserver in a way obvious to one skilled in the art. An example would beU.S. Pat. No. 6,697,838 Method and System for Annotating InformationResources in connection with browsing, in both connected anddisconnected States which enables someone browsing the web to createnotes, link them to the displayed web page and automatically retrievesuch notes on returning to the web page in question. Such persistence isalso demonstrated in the implementations, for example, ofwww.shiftspace.org or www.mystickies.com. In such cases the annotationcontent is stored separately from the content being annotated and theannotation system is functionally independent of the web server system.

Despite such existing systems there remains a major issue which untilnow has had no general solution for such functionally independentannotation systems: the problem of semantic integrity. For annotation tohave value its meaning must be preserved and in many cases this meaninglies not just in the content of the annotation, but in its relationshipto the underlying content. In some cases that relationship is verygeneral as when the annotation relates to the page as a whole. Anexample of this might be a sticky notelet with a text comment about thepage. But in other cases the user may wish to annotate not the page ingeneral but a specific element, or set of elements, on the page. Anintuitively obvious example of this is text highlighting. The yellowhighlighting annotation has meaning in relation not to the page but to aspecific block of text. This is one instance of a much more generalissue. We can envisage countless more styles of annotations a user mightwish to make in which the relationship between the annotation contentand the underlying page content is an integral part of the meaning. Insuch cases users will place annotation at a specific location for goodreason. An arrow points to a specific word or image not to the page ingeneral. A specific part of an image is circled. A user paints a pictureof a tail onto the rear end of a dog. In such cases the meaning of theannotation is very sensitive to its location on the page. It is thechallenge of ensuring that this relationship between annotation andcontent is preserved despite transformation of the page so that the truemeaning of the annotation is preserved that is the problem of semanticintegrity.

It might seem that the provision of semantic integrity is a trivialproblem which may be readily solved by saving the annotation's positionon the page. If a user were annotating a book this would work becausebook pages do not change. Web pages do change however and such changescan easily render annotations meaningless. For this reason the obvioussolution does not work. Even in the early days of the World Wide Web,when most pages were static and their content did not change a greatdeal, problems of semantic integrity could easily arise. For example, ifa user annotated a web page and that page was long enough to requirescrolling to display page content, then the annotations become spatiallymisplaced. If the page employed fluid rather than fixed display so thatwhen the browser display was resized the page layout changed, then theannotations become misplaced. Fluid display sites are not exceptional.An example of a popular such website is www.wikipedia.org. Furthermore,the same web page may be rendered differently by different platforms,and by different web browsers on the same platform, resulting inchanging the rendered location of elements on the page. And of course,sometimes the publisher would change the page content.

With today's web, often referred to as Web 2.0, the problem iscompounded because today's web is characterized by many technologiesthat result in pages changing continually. A large and growingpercentage of today's web pages are implemented with a database drivendesign whereby updated content is pushed onto webpages, composed thenrendered in real time. The web page retains its URL but its contentchanges regularly. Personalization may mean the page accessed by thesame URL being displayed differently for different users. Pages may notonly change from one instance of a user accessing the page to anotherbut may even change whilst the same user is viewing the page. A highprofile example is the newsfeed on www.facebook.com where a sub-sectionof the page, a nested viewport, scrolls continually. In this case anyannotation that comments on an element of the newsfeed will quicklybecome meaningless as that item drops down the list of updates and thendisappears from the screen. Another example is an annotated element of ascrollable region which is initially off-screen and then moved on-screenby page embedded action code; such an annotation will not be displayedas the element appears on the screen. Such dynamic behavior of web pagesmeans that it is impossible to view today's web pages as staticresources, their content must be assumed to be subject to continualchange.

Furthermore, semantic integrity has temporal as well as spatialdimension. When an annotation appears can be just as important as whereit appears. For example, if a user annotates a GIF image, but theannotation is only relevant for specific frames in the whole animation,then those are the frames (and the only frames) where the user wouldwant the annotation to appear. The problem of temporal semanticintegrity has become significant because web pages increasingly includenot just static media types such as text and image but dynamic mediatypes in which a timeline is important, for example audio and video. Inany attempt to annotate a piece of such dynamic content the ability toaddress elements on a timeline and provide temporal semantic integrityis crucial. Furthermore, because as described above, an increasingpercentage of pages are themselves dynamically updated by such means asin-page javascript, and because of timing delays caused for example bynetwork lag, the state of such dynamic media types as audio and videocan change in unpredictable ways that a naive treatment of the mediatimeline cannot reliably deal with.

The problem has become acute. Changes in page content and media with atemporal dimension are now so prevalent that without the ability tomaintain semantic integrity the provision of web page annotationsoftware is a license to create annotations that quickly appear brokenor meaningless.

One approach to this problem is to restrict what users can do in the wayof annotation. Prevent page layout problems due to resizing the browserwindow by enforcing a standard window size for annotation. Only allowannotation of pages with static design. Address content change by onlyallowing annotation styles less likely to quickly become meaningless,for example stickies which in some sense comment on the whole pagerather than a page element. Don't allow annotation on media types withtricky timing issues. Analyse the structure of the underlying documentand use this structure as a proxy for the on screen spatialrelationships that the user cares about. Take a copy of the page to beannotated and annotate that copy. But such approaches are not foolproofand severely limit the ways in which users may annotate and copyingraises potential copyright concerns. Hence such attempts fall far shortof the ideal.

Due to this unresolved problem of semantic integrity today's annotationtools are weak. Annotations such as the notelets of www.mystickies.comthat float above the whole page aren't specific enough for manyapplications. Users should be able to annotate specific page elements,and regions or points on such a page element, not just the page atlarge. Text highlighting is a step in the right direction but it is asmall step indeed, as is associating notelets with such highlighted textblocks. Web page content is increasingly media rich and it is naturalthat user should wish to be able to annotate all page media types withcontent of any media type and be assured of spatial and temporalsemantic integrity. The ideal is for users to have access to tools whichcan generate annotation content in any media and for those annotationsto deliver semantic integrity where possible and graceful degradation ifchanges to the page are so radical that it is not possible; to be ableto collaborate on such annotation creation both synchronously andasynchronously; and to be able to generate such content in a frameworkwhich can be extended to address unforeseen annotation requirements andmedia types.

Finally, existing systems do not comprehensively support annotations asprogrammable objects which may be accessed and modified not just byinteraction with human users, but by other computer programs, through awell defined API. The provision of such an API is important because itenables programmers to develop applications that can create and modifymultimedia annotations with semantic integrity.

As the description above makes clear, existing annotation systems have avariety of shortcomings As the description below will make clear,various embodiments of the present invention overcoming theseshortcomings and provide a variety of other advantages.

A prominent aspect of today's computing environment is networkedcomputing. Client computing devices are connected over the Internet oran intranet to each other and to server computers. Such client devicesinclude PCs running Microsoft Windows or Macs running MacOS, mobilephones running the Android operating system or the Apple iOS operatingsystem, tablet computers such as Apple Computer's iPad running the iOSoperating system, tablets running Android, or netbooks running Google'sChrome OS. Such server devices include computers running web serversoftware such the Apache Foundation's Apache or Microsoft's InternetInformation Server; database software such as Oracle's MySql orMicrosoft's SQL Server; and application specific applications serversoftware such a programs written in Ruby or Java.

In such an environment, computers issue requests for resources andservers respond to such requests and forward the requested resource. Acommon such resource is documents stored on servers and encoded in ascheme appropriate for consumption by a document client program.Commonly the client will translate such encoded documents into a formatwhich may be rendered and then displayed.

One important instance of such network computing is the World Wide Web,commonly abbreviated to WWW or W3 and commonly known as “the web.” Inthis case the document is a web page. Users use a web browser to viewweb pages which may well incorporate rich media and which typicallyinclude hyperlinks by means of which users may navigate between pages.The web is therefore a system of interlinked web pages.

Web pages are documents encoded in a markup language—a language with asyntax which adds structural information to a document by means ofspecial character sequences, hereinafter the ‘markup’. The markuplanguages typically used to structure documents (web pages) on the webare the HyperText Markup Language (hereinafter ‘HTML’) and the similareXtended HyperText Markup Language (hereinafter ‘XHTML’). XHTML is astricter form of HTML with consistent rules about the format of markupand the nesting of document elements and both are derived from SGML(Standards General Markup Language) and the SGML derived XML (eXtensibleMarkup Language) which is widely used as a portable method for encodingdocument structure and encapsulating content. Web page structure(hereinafter ‘structure’) is specified by the markup which identifiesthe start and end of individual components of the document. Thesecomponents form HTML/XHTML elements (hereinafter ‘elements’) and thepage content is contained within the elements.

XML is a text based language, thus the content of an XML element isalways text. However, the text content may be encoded to represent anymedia type, including, but not limited to images, audio, video.Furthermore, the text may contain an identifier used to access theactual content stored elsewhere; for example by using the UniformResource Locator (URL) to specify the location of the media data and themethod for retrieving it.

For the sake of simplicity hereinafter we shall refer to documents thatmay be XML, HTML or XHTML in the context as ‘XML documents’ andsimilarly to document elements that may be XML, HTML or XHTML as ‘XMLelements.’ Furthermore, we shall refer to ‘HTML or XHTML’ as ‘HTML’where the two are interchangeable for practical purposes.

Web browsers can issue requests for web pages. Servers respond with aweb page either taken directly from a local file specified by therequest or encapsulates the output of an application componentreferenced by the request. Such requests and responses typically use theHyperText Transfer Protocol (hereinafter ‘HTTP’), a client/server basedprotocol for requesting whole or partial documents.

A widely used technique for the implementation of applications thatprocess XML documents is to first parse the XML structure into anin-memory representation using the Document Object Model (hereinafter‘DOM’). The DOM is well known to those skilled in the art as across-platform and language independent means of representing andinteracting with objects in XML documents. The DOM has an applicationprogramming interface (hereinafter ‘DOM API’) by means of which programsmay interact with and process well formed and hence valid suchdocuments. The DOM is a tree structure which mirrors the XML documentstructure; nodes on the DOM tree represent XML document elements, theirattributes and content. Thus, objects within the DOM representationcorrespond to XML elements and their content—hence, the XML element isoften described, by those skilled in the art, as an XML object.

With the DOM API, programmers can build documents, navigate theirstructure, and add, modify, or delete elements and content. Anythingfound in an XML document can be accessed, changed, deleted, or addedusing the DOM API. Those skilled in the art will recognize the almostuniversal use of the DOM and DOM API for manipulating XML documents.

Those skilled in the art will recognize that a web browser is a specificimplementation of an application which uses the DOM API to parse XMLdocument structure in order to access and transform its content.Furthermore, web browsers are open platforms which support the extensionof their functionality via third party components. The interface andprotocol specifications to provide access to the inner operation of thebrowser, (which access is necessary to develop such extensions) ispublished and freely available to developers. Examples of extensionsare: support for new media types; customised web page display behaviour;additional menu options and web page development tools.

FIG. 1 is an exemplary high level block diagram of a system 100 enablingthe storage, retrieval, transmission, processing and display ofdocuments The components of the system are enclosed in the dotted line105. A high level description of a typical operation (document retrievaland display) using this system 100 follows. A user on a client computer110 interacts with a web browser 120 and by interaction with thatprogram generates a request 130 for an XML document 180 stored in adocument store 160 on a document server 150. The request 130 istransmitted over the communication channel 140 to the document server150 storing the document. Those skilled in the art will appreciate thatthe communication channel 140 may for example be the interne, anintranet or in the case wherein the client computer 110 and the documentserver computer 150 are the same, an internal system bus. The documentserver 150 locates the requested document 180 in its document store 160and transmits it over the communication channel 140 as a response 170 tothe client computer 110 and thence to the web browser 120. The webbrowser 120 parses the encoded document 180, processes it into a formatsuitable for representation in the DOM and inserts this formatted datainto the DOM by means of the DOM API. The rendering engine module of theweb browser 120 operates on the resultant DOM data structure(s) tocreate a rendered version of the document 190. This version of thedocument 190 is manifested.

Those skilled in the art will recognise that whilst this is only one usecase and it is simplified for the purposes of exposition. It isnonetheless representative of the key components and processes of thedocument system.

The embodiment of the ‘annotation system’ (hereinafter ‘program’)comprises two components a) an annotation server, and b) an annotationclient. These components are distinct from those of the document system.As the annotation system exists to annotate documents it presumes theprior existence of a document system.

The embodiment below is set in the context of the web. The annotationclient is implemented as an extension to the user's web browser(hereinafter ‘browser extension”) and the annotation server is a serverhosted collection of support functions (hereinafter ‘server’). Such abrowser extension component may be referred to as a ‘plugin’ or an‘add-on.’ An example of a programming language used to extend browserfunctionality is JavaScript. Examples of server software hosting suchsupport functions would include a database management system such asMySQL or PostgreSQL and programs written in languages such as Java orRuby with (as required) support from an application framework such asStruts (for Java) or Rails (for Ruby).

FIG. 2 is a high level block diagram of a document system 100, whosecomponents are enclosed in the dotted line 105 and an annotation system200, whose components are enclosed in the solid line 205. The annotationsystem 200 and the document system 100 share some resources: the clientcomputer 110, the communication channel 140 and the web browser 120. Theannotation system 200 has resources the document system 100 does not:the annotation server 250 with its annotation store 260 and the browserextension 285. The document system 100 has resources the annotationsystem 200 does not: the document server 150 with its document store160.

The annotation system 200 assumes the existence of a document system 100and enables the storage, retrieval, transmission, processing and displayof annotations on documents served by the document server 150. A highlevel description of a typical operation (annotation retrieval anddisplay of an annotation 280 and its display on its associated document180 served by the annotation system 100) using this system 200 follows.

A user on a client computer 110 interacts with a web browser 120 andwith the browser extension 285 and by that interaction generates arequest 230 for an annotation 280 stored in an annotation store 260 onan annotation server 250. The request 230 is transmitted over thecommunication channel 140 to the annotation server 250 storing theannotation. The annotation server 250 locates the requested annotation280 in its annotation store 260 and transmits it over the communicationchannel 140 as a response 270 to the client computer 110 and thence tothe browser extension 285. The browser extension 285 parses theretrieved annotation 280 and processes it into a format suitable forrepresentation in the DOM. As will be outlined in detail below, thebrowser extension 285 then inserts this processed data into the DOM bymeans of the DOM API thereby creating a composite of the document to beannotated 180 and the annotation 280 which is annotating document 180.The rendering engine module of the web browser 120 operates on theresultant composite DOM structure to create a rendered version of thecomposite, document 290. This version of the document 290 is displayed.

It will be recognized by those skilled in the art that notwithstandingshared components the document system 100 and the annotation system 200are functionally independent. This point will be discussed in greaterdetail below.

Whilst the exemplary implementation describes a browser extension moduleand a particular division of labor between the extension module and theserver modules it will be recognised by those skilled in the art thatthe functionality of browsers themselves continues to evolve as doestechnology to support browser extensions and server software. Theinvention may, for example, be implemented using any mechanism whichsupports extension of browser functionality such as Google's “NativeClient” (NaCL). Furthermore, in an alternative implementation, thefunctionality of the extension may be incorporated into the core browserin its entirety. Indeed, as is obvious to one skilled in the art, in thelight of such ongoing technical change, the division of labor betweenthe browser, the browser extension and the server is itself subject tochange and the program may therefore take the form of many embodiments.The functionality of the program may, for example, be heavily weightedto the server with a thin client or alternatively may, for example, beheavily weighted to the client with the server simply storingannotations created and edited by interaction with the client.

The invention enables powerful and novel annotations on web pages. Webpage content may be as simple as text or it may be complex media such asvideo. XML elements specify how and where visible content is rendered onthe browser's display area and how non-visible content is actioned e.g.how and when audio is played. These XML elements with their content forma rich set of objects. The content of XML elements are of various types.Hereinafter we refer to such types simply as ‘media types’.

Web pages are structured compound objects that include objects of manymedia types. Many such objects are directly included, for example textand images with a data URI, but others may be included by reference andall are subject to dynamic modification.

Page objects may have spatial and or temporal dimension. Visualmanifestations of page objects are spatially related to each other onthe page. Those skilled in the art will recognise that page layout maychange for many reasons and that as it does, for whatever reason, suchspatial relations may also change. Similarly, the manifestation of pageobjects are temporally related on the page and such temporal relationsmay also change.

In an intuitive sense, annotations may be said to be placed “onto” a webpage, thereby annotating the page. The location on the page to which theannotation is attached is the ‘annotation target’. In placing theannotation, the user specifies a relationship between the annotationcontent and the annotation target. This relationship has meaning to theuser and hence is a ‘semantic relationship.’ Recognizing that theimportance of this semantic dimension of annotation, an importantpurpose of the described embodiment is to capture the semanticrelationship and ensure that annotations are re-manifested with thesemantic relationship intact on subsequent instances of the web page.Components of the embodiment undertake analysis of page content andstructure to capture this semantic relationship. This analysis ishereinafter ‘semantic analysis.’ The property of preservation of thissemantic relationship derived from semantic analysis (despite changes inthe underlying document from one instance of its manifestation toanother) is hereinafter termed ‘semantic integrity.’

FIG. 17 illustrates the property of semantic integrity. The FIG. 17shows two instances of the same web page 2306 and 2316. Although theydiffer in layout and specific content, they are the same web page asthey result from a retrieval of the same URL. The first instance of theweb page 2306 shows the page comprising of four page objects, 2304,2307, 2308 and 2309. Examples of such page objects are text blocks orimages. Page object 2304 has been annotated. The annotation content 2303has been placed onto the web page 2306 in a specific relationship 2302to a point 2301 on the annotated page object 2304. This relationship2302 is a semantic relationship, determined by the user, as describedabove. The second instance of the same web page 2316 shows the page haschanged significantly: it now comprises three page objects, 2304, 2310and 2312. Only the page object 2304 remains from the earlier instance ofthe web page 2306 and this page object is now shown to be a differentsize and in a different location on the page 2316. In this secondinstance the same annotation content 2303 has been placed onto the webpage 2316 in the same specific relationship 2302 to the point 2301 onthe annotated page object 2304. Hence the semantic relationship,determined by the user, between the annotation content 2303 and thelocation on the page 2301 is maintained. This example of thepreservation of the semantic relationship created by the user, despitesignificant change in the web page being annotated is an example of theproperty of semantic integrity.

Content placed ‘on’ the page is what is typically naively thought of asthe annotation. Familiar examples of annotations are notelets and texthighlighting but the program enables annotation content of any mediatype or combination of such types. Whilst some annotations have a visualdimension e.g. notelets which are of type text or composite objects suchas widgets, other annotations may not have such a visualization e.g.audio with no visualized player. Recognizing such differences, we saythat different annotation media types are “manifested” differently. Forexample, visualizations such as images are manifested by being renderedand displayed, audio attached to a page with no visible player ismanifested by being played, video is manifested by being rendered andplayed, and links are manifested by being followed. The act ofmanifesting annotation content creates a ‘manifestation’ of theannotation content (e.g., a rendered and displayed image in the case ofimage annotation content). It is this manifestation which the userexperiences and it is such manifestations that the user manipulates toplace the annotation in relation to a target manifestation to establisha semantic relationship. In the case of visual manifestations this is adirect and evident manipulation. In the case of non-visualmanifestations it remains logically true, but the relationship ismanipulated by means of a user interface representing this relationship.

The program enables the creation of annotation content in ways thatrange from the familiar and simple, for example typing text into anotelet, to the rich, for example ‘painting’. The program allows for allsuch styles of annotation. Whilst some content is of very generalapplicability e.g. a notelet may be used to annotate practically anytype of page object, other content is much more specific and only makessense when applied to certain types of page objects e.g. a videoannotation tool may only be applied to video.

The data which the program generates to describe the annotation contentitself comprises one or two components (a) data required to manifest thecontent on the annotated web page, hereinafter the ‘manifestation data’and optionally (b) data which controls elements of how the manifestationbehaves; hereinafter ‘manifestation control data’ or ‘MCD’. Anannotation always has ‘manifestation data’ but may not have‘manifestation control data.’

The invention supports manifestation data of any type supported by theDOM. Annotation need not therefore be simple text or images but may be,for example, multimedia such as video, hyperlinks or form elements suchas text fields, checkboxes, buttons and selection menus. Script andembedded object elements are also valid content media types and, as willbe discussed in more detail below. Such dynamic elements within theannotation content may be used to implement ‘annotation behavior’ byinteracting with the user, responding to external events and generatingevents to influence the behavior of other annotations. The manifestationmay also be of combinations of such media types within an HTML fragment,e.g. a speech balloon image into which the user may input text or otherdata. In another example, the manifestation may not be visual at all,but may rather be an audio clip that is played. It is the manifestationof this manifestation data which may be naively considered to be theannotation.

The MCD is a specification of how the program manifests the annotation:it comprises

-   -   i. manifestation timing data—used to control the temporal        behavior of the manifestation e.g. how long the annotation        should be manifested after it is ‘started’ or the maximum delay        from loading of the annotation to the manifestation of the        annotation. Such an ‘MCD delay’ MCD delay applies to all        associations for that annotation;    -   ii. manifestation realization data—used to control the behavior        of an annotation. For example, to control the visual behavior of        an annotation with a visual manifestation e.g. the annotation        abruptly appears and disappears, or the annotation fades in and        out, the annotation may slide in from the edge of the browser        display. In a further example, to control the audio behavior of        an annotation with an audible manifestation e.g. to control the        volume fading in or out; and    -   iii. manifestation external interface specification data—used to        control how the annotation interfaces with external objects e.g.        how the annotation will respond to external events and what        aspects of annotation data may be exposed for modification by        other annotations.

In the event that an annotation has no MCD specification, a defaultbehaviour is implemented for the annotation:

-   -   the annotation is manifested immediately with no realization        modification    -   the annotation continues to manifest until the underlying web        page is changed    -   the annotation responds only to the Annotation API (as detailed        below), without behavior modification

In the described implementation the program uses an event mechanism toimplement operations on the annotation. Named events are sent to theannotation (e.g. ‘start’ event) and are interpreted according todeclarations in the MCD. A core set of events (e.g.: timers, annotationstart/stop) is generated by the program and annotations may define anddispatch their own events. Those skilled in the art will recognize thatthere are alternative mechanisms for annotation control, for example viadirect calls to an Annotation API with dynamic methods corresponding toevents consumed by the annotation.

The MCD may, in one example, include timing data specifying timinginformation about the manifestation, such as—display the speech balloonfor five seconds, then make it invisible for 5 seconds, then display itagain and so on (blinking). In another example, the external interfacedata may specify how the manifestation is to behave when a specifiedevent occurs, such as—if the user clicks on this element of the contentthen dim the speech balloon. Those skilled in the art will recognizethat by virtue of such control data the “same” content (i.e. the samemanifestation data) can behave differently.

The MCD Specification comprises key/value pairs, for example:

-   -   start delay: <seconds from ‘start’ event to manifest        commencement>    -   manifestation duration: <seconds to manifest to the annotation>    -   blink: <start/stop cycle, on time, off time>    -   fade in: <seconds over which manifestation linearly taken from        0% to 100%>    -   fade out: <seconds over which manifestation linearly taken from        100% to 0%>    -   expose: <annotation attribute exposed for modification>    -   consume: <event to action binding>

The user does not directly edit such key/value pairs. Such statementsare typically manipulated by program components which present a suitableuser interface. It will be recognized by those skilled in the arthowever that, subject to access control, such information may beaccessed and edited programatically by other programs. Those skilled inthe art will further recognize that there are viable alternative methodsof expression of such control data and the selection of such alternativewill not affect the viability of the embodiment.

FIG. 3 illustrates the use of the MCD to influence the timing of themanifestation of an annotation. The elapsed time of the manifestation ofthe annotated page object is represented by 310. The annotated pageobject is manifested at time 311 and continues to be manifested untiltime 319. The time interval 310 may vary in real terms from one instanceof the manifestation to the next. For example, if the targeted pageobject is a video clip, network lag may result in data streaming delays.The annotation is attached to a target on the annotated page object attime 313 on the elapsed time of the annotated page object 310 but theannotation has a manifestation delay ‘MCD delay’ 320 specified in theMCD. This delay applies to all associations for that annotation whichmeans that even when the target is reached at time 313 on the elapsedtime of the annotated page object the annotation does not manifestimmediately but after the period of the MCD delay 320. Only once the MCDdelay 320 is exhausted at time 315 on the elapsed time of the annotatedpage object 310 does the annotation manifest. The annotation continuesto manifest until it terminates at the manifestation end point at time317 on the elapsed time of the annotated page object 310. Thistermination of the manifestation at time 317 on the elapsed time of theannotated page object 310 overrides the default behavior described above(to continue manifestation until the annotated web page is unloaded ormanifestation of the annotated page object ceases, whichever occursfirst). Such an override of the default behavior may, for example,result from an MCD specification of the duration of the manifestationinterval 330 or it may be determined by a ‘stop’ event communicated tothe annotation.

Hence it may be seen that the program supports both rich annotationcontent and powerful control over both the spatial and temporaldimensions of the manifestation of such content. Furthermore, as will bedetailed below, the program supports additional operations onannotations. They may, for example, be managed in sets, shared bygroups, browsed, searched, saved and restored with semantic integrityand every aspect of their content and behavior may be programmaticallyedited. For the program, therefore, an annotation comprises not only thecontent (which is generally naively considered to be the annotation) buta significant amount of additional information and functionality.

Notwithstanding the above, the text below may use the term ‘annotation’to refer to the manifestation of the annotation content when intuitivelyappropriate, thereby according with the naive use of the word.

The Annotation Point

To define the exact location of manifested content such content has tobe represented by a point, (hereinafter the ‘annotation point.’) This isa known point on the annotation manifestation (spatial and/or temporal)which is used to represent the annotation manifestation's location inspace and/or time.

Spatial annotation points will typically have an intuitive location ‘on’the annotation manifestation. An example of such a point is the tip of aspeech balloon. In the second example of an annotation such as a circle,the annotation point could be the center of the circle and by placingthe annotation point ‘on the page’ one is, in effect, placing the circlein a precisely determined position on the page in relation to the pageobject being annotated.

Temporal annotation points are used to trigger events which influencethe behaviour of the annotation. They are associated with temporalaspects of a target and may be identified by timer information whichspecifies a duration on the timeline of the page object containing thetarget e.g. 5 seconds after start of audio playback. They mayalternatively be identified by recognition of information in the contente.g. an image on frame 57 of movie playback. The standard temporalannotation points are ‘start’ and ‘stop’. Hence, temporal annotationpoints may identify the ‘start’ and ‘end’ of the manifestation of theannotation content. The ‘start’ annotation point is the trigger to startthe annotation, for example, to render an image or to start playing anaudio clip; the ‘stop’ annotation point is the trigger to stopmanifesting the annotation, for example, to hide a visible annotation orto pause an audio clip.

An annotation may be associated with multiple targets, each of whichreferences a different annotation point. Such annotation points maytrigger multiple ‘start’, ‘stop’, ‘start’, . . . events which will havethe corresponding effects in a time order specified by evaluation of thecorresponding annotation associations discussed below. A temporalannotation point sequence of two ‘start’ triggers, or two ‘stop’triggers, causes the second to have no effect (i.e. the annotationcannot be started again). These triggers are delivered to the annotationvia annotation events and are interpreted according to specifications inthe MCD. The default behaviour of the ‘start’ event is as describedabove, similarly for the ‘stop’ event. If there are no temporalannotation points specifying a ‘start’ event, then the program suppliesone at the instant the annotation is selected for display. Temporalannotation points may be used to trigger any event exposed by a MCD‘consume’ specification and hence to generate more complex behaviorsthan annotation start and annotation end.

In the case where an annotation with temporal annotation points, isselected for manifestation after any related target timeline has passedthen one of the following recovery actions is executed: (1) if thetarget is timer based (e.g. 5 seconds after start of audio playback)then the program actions the associated annotation by sending the startevent immediately; or (2) if the target is derived from an analysis ofcontent then a dialog is presented to the user to ask if they wish toredisplay the page and hence replay the content; if the user declinesthe restart operation then the annotation start is not actioned

Because the annotation point references the content multiple annotationpoints may reference the same content.

The Annotation Target

The program enables a user to place the content on the page and toanchor it onto a specific location on the page—the target. This targetmay be determined by a human user by interaction with the program orspecified by a computer user programmatically by accessing an API (seebelow). The target is one of: the web page itself; a web page object, acollection of related web page objects, a region of a web page object; aregion which spans a collection of related web page objects or aspecific point within a web page object. (A ‘region’ is a section of anobject where the start of the section does not necessarily coincide withthe start of the object and the end of the section does not necessarilycoincide with the end of the object.) Those skilled in the art willrecognise that collections of related web page objects will typically bea sub-tree of the web page document.

All targets have a manifestation. The invention supports a broad rangeof targets (any media types addressable by the DOM) hence allannotations may be attached to targets of any such type so long as suchan attachment is meaningful.

Annotation targets may be spatial and/or temporal. Hence annotations maybe attached to a specified location on a visualized manifestation and/orattached to a manifestation at a specified instant during itsmanifestation. Hence an annotation may be said to have spatial and/ortemporal dimension.

Examples of such targets are the whole page, a block of text, an image,a group of pixels in an image, a movie, a set of frames in a movie, asingle frame in a movie or a point on such a frame. A collection of pageobjects comprising a sub-tree of the document structure may also beselected as a target, for example: a whole table, a single table row, orall XHTML elements enclosed within a <div>. Sub-trees as targets are ofparticular use where the program has used specialist knowledge of pagesfrom a website to identify semantically associated large sections of theweb page document—e.g. entries in a news feed. The target may also benon-visual, for example a point on the timeline of an audio track.

The target is defined by the program by means of analysis of the pageobject(s) content and optionally by its relationship to other objects onthe page. The data generated by such analysis is used by the program tofind the target again in subsequent instances of the web page and ishereinafter called ‘Target Locating Data’.

Despite the fact that a target with a visualization may often be set bythe user to be an area (page, object or region), just as the programuses the annotation point to represent the annotation content, so doesit use a point in this target area to represent the precise location ofthe target. This point, hereinafter ‘target point’, may be a programdefault, determined from the type of target or it may be specified bythe user. If the user specifies a point as a target then this is fixedsuch that target=target point. In the case where the user specifies apage object or region as target, information about the target area isstored and the program calculates an appropriate target point which isre-calculated should any of the target context change. For example, thepoint for a text phrase that occupies a single line on the display maybe calculated as the centre of the phrase; if the display is resized sothat the phrase now occupies multiple lines, the point must berecalculated and may now reference the centre of the display areaoccupied by the phrase. The target point for page objects with novisualisation (e.g. audio) is the page origin.

In the case of page content that has a temporal aspect (e.g. audio andvideo), the default temporal target is the first instant that the pagemanifests (but this may be overridden to be any point on the playtimeline).

The invention's very general method and process enabling users tospecify the target to a desired degree of detail is important. Firstly,a user may want to annotate a very specific piece of content on the pagerather than the whole page or all of a page object. Targets referencinga collection of related objects (e.g. a document sub-tree) allow theuser to annotate content spread across multiple page objects and which‘mean’ something on the web page. It may be, for example, that he wantsto draw attention to a couple of lines in a large text block or an entryin a newsfeed or a particular phrase in a piece of music or a specificlocation on a specific frame of a video. Because the target may bespatial and temporal the user may, for example, place a visualannotation on a specified point on a specified frame of a video.Secondly, as we shall see below, this capability is critical for theprovision of semantic integrity.

The Annotation Association

The annotation association links the manifestation of the annotationcontent to the manifestation of the target by specifying one or morerelationships between the annotation point and the target point and itthereby represents the semantic relationship described above. As alsonoted above, there are many relations, spatial and temporal, betweenpage objects. The association however is unique in that it is the onlysuch set of relations representing the user determined semanticrelationship between the manifestations of the annotation content andannotation target. The relations specified in the association determinethe offset between the annotation content and the target in space andtime. Each association may have one or both of such spatial and temporalrelations. (Whilst it is always more accurate to speak of theassociation between the manifestations of the annotation content and thetarget we will from time to time speak more intuitively of theassociation being between the annotation content and the target, orbetween the annotation and the target.)

The association with spatial dimension (hereinafter ‘spatial offset’)notes the distance and direction from annotation point to the targetpoint, thereby determining where the manifested content is in relationto the target. The spatial offset has a default which may be zero, inwhich case the annotation point is co-incident with the target point inspace. In future instances of the web page, the spatial offset enablesthe program to manifest the content in the same spatial relationship tothe target as in the first instance of the document.

FIG. 4 shows how the position of manifested annotation content on thepage is determined by its spatial offset from the target. Web page 410contains a visually manifested page object 420. Examples of such pageobjects are text blocks and images. The annotation point 440 is relatedto the target point 430 by the spatial offset 450. If user interactionalters the position of the annotation then the annotation point 440 willchange Similarly, if user interaction alters the position of the targetthen the target point 430 will change. Changing the annotation point 440changes the spatial offset 450; changing the target point 430 changesthe annotation point 440 to maintain the spatial offset 450.

Given the target point and the spatial offset the annotation point maybe located precisely and hence a second instance of the annotation maybe manifested precisely in relation to the target. It will be understoodby those skilled in the art that the spatial offset directly representsthe semantic relationship determined by the user between the annotationcontent and the target. It will further be understood that non-visualannotation content (e.g. audio) has no spatial annotation point and nospatial offset.

All page objects with a visual manifestation are spatially related onthe page. The spatial association is a special such relation—thatbetween annotation point and target point.

Some page objects, such as audio or video players, manifest content witha timeline. Such content may have annotations attached to it at anypoint or points on its timeline. The association with temporal dimension(hereinafter ‘temporal offset’) notes when the annotation is manifestedtemporally in relation to a target. More specifically it specifies thedifference between the instant on the page object timeline representedby the target and the instant on the page object timeline that theannotation is signalled to start. By default, the temporal offset iszero in which case the annotation point is co-incident with the targetin time. In future instances of the web page, the temporal offsetenables the program to manifest the content in the same temporalrelationship to the target on the page object timeline as was the casein the original instance. (As noted above, the temporal offset isdistinguished from the MCD delay in that the temporal offset applies toa single association whereas an MCD delay applies to all associationsfor that annotation.)

FIG. 5 illustrates the use of the temporal offset to influence thetiming of the manifestation of an annotation. The elapsed time of themanifestation of the annotated page object is represented by 510. Theannotated page object is manifested at time 511 and continues to bemanifested until time 519. The time interval 510 may vary in real termsfrom one instance of the manifestation to the next. For example, if thetargeted page object is a video clip, network lag may result in datastreaming delays. The annotation is attached to a target on theannotated page object at time 513 on the elapsed time of the annotatedpage object 510 but the annotation has a temporal offset 520 specifiedin the annotation association data. This temporal offset 520 means thateven when the target is reached at time 513 on the elapsed time of theannotated page object the annotation does not manifest immediately butis delayed for the period of the temporal offset 520. Only once thetemporal offset 520 is exhausted at time 515 on the elapsed time of theannotated page object 510 does the annotation manifest. The annotationcontinues to manifest until it terminates at the manifestation end pointat time 517 on the elapsed time of the annotated page object 510. Thistermination of the manifestation at time 517 on the elapsed time of theannotated page object 510 overrides the default behavior described above(to continue manifestation until the annotated web page is unloaded ormanifestation of the annotated page object ceases, whichever occursfirst). Such an override of the default behavior may, for example,result from an MCD specification of the duration of the manifestationinterval 530 or it may be determined by an event communicated to theannotation. Such an event may be triggered by a timer (measuring aninterval from annotation ‘start’) or by content manifested on the pageobject timeline. By such means those skilled in the art will appreciatethat the program allows a user to, for example, attach a voice comment 5seconds from the start of a video clip or to attach an image to frame157 of a video clip, and for those annotations to persist until eithertheir duration as specified in the MCD elapses or an event signals theannotation to stop.

As those skilled in the art will recognize, and as noted above, temporalmanifestations are affected by operating system, network and browserload; for example, an audio track that should start as soon as the webpage is loaded may be delayed due to poor network performance. Atemporal constraint specifies the limits on delay between targettimeline point and annotation manifestation; for example the annotation‘start’ event can be suppressed if more than a specified number secondshave elapsed since the target was manifested. Due to the possibility oflag it will be appreciated that attaching annotations to media of theform ‘start annotation manifestation X seconds after page objectmanifests’ may well result in annotations appearing at the wrong time,but that attaching annotations to targets embodied in the contentovercomes this problem of annotation manifestation.

All page objects are temporally related. The temporal association is aspecial such temporal relation—that between annotation point and targetpoint.

Some annotations may be moved spatially or temporally in relation totheir targets, subject to constraints on the offset (hereinafter ‘offsetconstraint’). Such constraints may be implemented either manually by theuser or automatically by the program. An example of a constraint is amaximum distance from the target in space or time. Constraints may varydepending upon the style of annotation. Some annotation content mayrequire pin-point accuracy with respect to its target (e.g. ‘pin thetail on the donkey’) whereas others may be offset with a considerabledegree of flexibility (e.g. a notelet type text annotation). Theinvention supports multiple constraint policies and different annotationcontent may be constrained in different ways.

If the annotation content manifestation has a visualization and isexactly the same size (occupies the same display screen area) as thetarget and is co-incident with the target, then the target is covered bythe annotation content and therefore disappears. This allows theeffective replacement or apparent editing of web page objects orregions. In a similar manner non-visual manifestations such as audio maybe apparently edited by removing underlying audio from the DOM thenmanifesting the replacement audio in its place. Because spatial andtemporal offsets may edited and thereby updated, animations may becreated by such updates to the offset and hence to the position in timeand/or space of the manifested content.

The Annotation Triple

As noted above, for the program an annotation is not synonymous with theintuitive notion of an annotation as its manifestation content. Anannotation comprises a triple of content, target and the associationthat determines the spatial and/or temporal relationship between thecontent and target.

The data defining an ‘annotation triple’ (uniquely identified by its‘triple identifier’) is the combination of ‘annotation point data’,‘target locating data’ and ‘association data’.

The annotation point data is a reference to the annotation content data.Multiple triples may reference the same annotation content data. Itfurther specifies a point on the manifestation of the annotation contentwhich is used to generate the association.

The target locating data is that data generated by the program when anannotation is created or edited on a given document and which isdesigned to enable the program to find the target in a subsequentinstance of that document. Such data enables a process whereby a targetmay be identified. Note that target locating data is not necessarilyidentical to the physical location of the target. Rather the programuses the target locating data to generate the target point which is aunique spatial and/or temporal location on the target. Hence, a target'sposition on a page may differ from one instance to the next butnotwithstanding this change the target locating data is designed toenable the program to find the target location.

The association data is that data representing the spatial and/ortemporal relationship between the annotation and the target. Morespecifically it specifies the spatial and/or temporal relationshipbetween the annotation point and the target point by means of thespatial and temporal offsets.

FIG. 6 represents the annotation data structure as described above. Theannotation data 600 comprises the annotation triple data 602, uniquelyidentified by the unique identifier 610, and the annotation content data650 which is included by reference 626. The annotation triple data 602itself comprises the annotation point data 620 comprising spatial data622, temporal data 624 and annotation content reference data 626, theassociation data 630 comprising constraints data 632, spatial offsetdata 634 and temporal offset data 636 and the target locating data 640comprising spatial data 642 and temporal data 644. The annotationcontent reference data 626 references the annotation content data 650which comprises the manifestation data 652 and the manifestation controldata 654. Multiple annotation triples 602 may have the same value forthe annotation content reference data 626 and thereby share theannotation content data 650 in which case the set of such annotationtriple data 602 together with the shared annotation content data 650comprise the annotation data 600.

We may now consider some simple examples. The annotation content couldbe a notelet which is associated with a target which is an image on theweb page. In another example, the content could be an audio clip, thestart of which is associated with a specific frame of a movie clip onthe web page. In a more complex example the annotation point of a speechballoon references the annotation content (manifestation data requiredto render the balloon and MCD data specifying that the content shall bedisplayed for five seconds and then vanish) and the association linksthe annotation content to the target spatially and temporally with zerooffsets. Hence, the annotation point will be coincident with the spatialtarget and the annotation start will be coincident with themanifestation of the target start event. If the target for the speechballoon manifestation is the center of the movie display space and it isto be manifested starting at frame 198 then when the movie plays, theprogram identifies frame 198 and dispatches the annotation ‘start’event. The annotation is manifested in accordance with the MCD. Hencethe speech balloon is displayed in the center of the video display areafor five seconds then becomes invisible. By editing the association andthe MCD the balloon's position on the page and the time it manifests maybe altered.

The program allows for annotations with multiple annotation triples.This is useful for content such as arrows that have a beginning and anend. Furthermore, multi-component annotation content may be chained froma single target, each with their own associations and constraints. This,for example, would be appropriate for the display of a note referring toa particular word or phrase in a text target—the first content is anarrow graphic, the second is the note itself. As the note is moved, thearrow graphic is automatically re-drawn to join the target to the note.

FIG. 7 shows a multi-component annotation 700 on web page object 720.Annotation_1 710 has annotation point 712, target 714 on annotation_2720, and spatial offset 716. Annotation_2 720 has annotation point 722,target 724, and spatial offset 726. Annotation_2 720 is attached to thepage object being annotated 720 and annotation_1 710 is attached toannotation 2. Together annotation_1 710 and annotation_2 720 comprise amultipart annotation attached to the target 724 on web page object 720.

Additional annotation triples may be defined to create a purely semanticdependency with content elsewhere on the web page. These triples, forexample, would be appropriate where an annotation on one page object isrelevant only if other page object(s) are present and contain specificcontent.

An annotation may have multiple triples, each with its own uniqueidentifier, with the same content referenced by all the triples, bymeans of the annotation point. Where an annotation has multiple triplesand, hence multiple spatial associations each with spatial constraints,in the event that the targets are manifested in locations which wouldcause one of more of the constraint checks to fail then the annotationis not manifested. It is marked as inactive and may not be started byany means. Should the page content change, for example as the result ofin-page JavaScript execution, such that all spatial constraints may besatisfied, the annotation is manifested.

Annotations

The data for an annotation consists of all its annotation triple datai.e. the group of triples, plus the annotation content data. The contentdata is in this way shared between the one or more triples that aremembers of the annotation and the annotation has a unique groupidentifier.

Annotation Sets

A page may be annotated with one or more annotations intended to be seentogether (hereinafter the ‘annotation set’). The layout of multipleannotations displayed simultaneously on the same page is subject toconstraints and layout policy. As will be understood by those skilled inthe art, multiple layout policies may be realised by groupingannotations into collections for the purpose of applying such a policy.For example, there may be one collection for all annotations attached tothe page and another collection for all annotations attached to pageobjects targets. Any number of annotations and annotation sets may beapplied to any web page. Each annotation set is saved to the server as adiscrete item.

Each annotation is assigned a unique identifier and its position withinthe ordering of all annotations in the set determines the sequence andlayering precedence for visual annotations. Saving such information foreach annotation fixes these relationships and allows for laterrestoration to recreate the same manifestation in the same relationshipto the target (or a manifestation which whilst not exactly the same isthe same subject to defined parameters). Such ordering and layeringinformation is an example of an ‘Annotation Set Attribute’. Theseattributes will be discussed in more detail below.

The total of such data required to describe the annotation set (but notthe Annotation Data of the annotation members of the set) is hereinafterreferred to as the ‘Annotation Set Data.’

As used herein, data representing an annotation set may directly includedata describing the annotation set members or may reference the memberdata stored elsewhere.

Annotation Programmability

Annotations and annotation sets may be edited by means of a programmaticinterface. The Annotation Data API provides access to all datastructures in an annotation i.e. the Annotation Data. Programs which areallowed access to this API may thereby programmatically control allaspects of the annotation.

FIG. 15 illustrates the effect of editing elements of the annotationdata triple by means of the Annotation Data API. The original annotationconsists of content 2106 with association 2110 to target 2108. Thetarget 2108 is on page object 2104 on web page 2102. It is possible toedit the content 2106, the target 2108 or the association 2110programmatically. In the case where the content 2108 has been edited thenew content 2120 is manifested but nothing else is changed—the positionof the new content 2120 with respect to the target 2108 as defined bythe association 2110 is identical. In the case where the target 2108 hasbeen edited the new target is 2130; the content 2106 is unchanged butbecause the association 2132 is unchanged, the position of the content2106 changes. In the case where the association 2110 has been edited tothe new association 2140 the position of the content 2106 is changedaccordingly.

The Annotation Set API provides access to all data structures describingan annotation set i.e. the Annotation Set Data. Programs which areallowed access to this API may thereby programmatically control allaspects of the annotation set. These APIs together comprise theprogram's Application Programming Interface (hereinafter ‘AnnotationAPI’). By means of this API all the data structures defining anannotation set, and all data structures defining all annotation membersof the set, may be accessed and edited. Specific annotation data may beprotected against view and/or update access via ‘expose’ specificationsin the MCD.

The Annotation API may be used by active components (e.g. JavaScriptscript elements) within an annotation manifestation data to modify otherexisting annotations within an annotation set. Thus, it will berecognised by those skilled in the art that the Annotation API providesa means by which the program may receive annotation input datarepresenting annotation set content for annotating targets in web pages.

Hence, as will be appreciated by those skilled in the art, theannotations described in the invention are programmable. Editing andthereby changing the annotation data may, for example, move theannotation's location (in time and/or space) in relation to the target,make it visible for a shorter time or change its layering with respectto other annotations. It will further be appreciated that, subject toaccess control, programs other than those described below may be writtenwhich utilize the Annotation API to program annotations. For example, itis perfectly possible to write an application for an Android mobilephone which could alter an annotation's stored data and hence anannotation's manifestation or the behavior of the manifestation; or forJavaScript embedded within a web page to be aware of annotations placedonto the page.

Annotation Temporal Behavior

From the above description it will be recognized by those skilled in theart that the temporal behavior of the annotation can be specified by twomechanisms: the association temporal offset and the MCD timing data.Thus, there are two ways an annotation can be manifested with an initialdelay, i.e. by means of:

1. the annotation triple: the target manifests and the association linksto the ‘start’ event with delay specified in the association temporaloffset, and

2. the MCD: the MCD specifies a start delay.

It is also possible for an annotation to employ both mechanisms.

3. the triple association links to ‘start’ event with a temporal offset(d1) and the MCD specifies a start delay (d2). The result of which isthat the annotation starts at d1+d2 after the target appears on theobject timeline.

FIG. 8 illustrates the use of both the ‘temporal offset’ and the ‘MCDdelay’ for controlling annotation temporal behavior. The elapsed time ofthe manifestation of the annotated page object is represented by 810.The annotated page object is manifested at time 811 and continues to bemanifested until time 816. The time interval 816 may vary in real termsfrom one instance of the manifestation to the next. (For example, if thetargeted page object is a video clip, network lag may result in datastreaming delays.) The annotation is attached to a target on theannotated page object at time 812 on the elapsed time of the annotatedpage object 810. In this instance however the annotation associationdata includes a temporal offset 820 so even when the target is reachedat time 812 the annotation will not manifest until the temporal offset820 is exhausted at time 813. At time 813 the temporal offset 820expires and the ‘start’ event is sent to the annotation. This annotationhas a manifestation delay ‘MCD delay’ 830 specified in the MCD. Thisdelay means that even when the start event is sent at time 813 theannotation does not manifest immediately but delays for the timeinterval of the MCD delay 830. Only once the MCD delay 830 is exhaustedat time 814 on the elapsed time of the annotated page object 810 doesthe annotation manifest. The annotation continues to manifest until itterminates at the manifestation end point at time 815 on the elapsedtime of the annotated page object 810 after an annotation manifestationinterval of 840. This termination of the manifestation at time 815 onthe elapsed time of the annotated page object 810 overrides the defaultbehavior described above (to continue manifestation until the annotatedweb page changes). Such an override of the default behavior may, forexample, result from an MCD specification of the duration of themanifestation interval 840 or it may be determined by the ‘stop’ eventbeing sent to the annotation.

It will be noted that, as described, there can be two ways to triggerthe ‘stop’ event. The first one actioned will cause the manifestation toend, subsequent ‘stop’ events have no effect. This is also true for thegeneral case of multiple ‘start’ or ‘stop’ events.

Using only the MCD to specify a start delay does not work in all casesas the delay is applied identically to all associations that link to the‘start’ event and there are circumstances when this may well not be whatthe user wants. Hence those skilled in the art will recognize that thedescribed architecture, which also supports temporal offsets, offersvaluable functionality.

Annotation Persistence

All data for annotation sets may be saved to the server so thatannotations may be searched, browsed, shared and retrieved andreconstituted on the annotated web page.

Annotation Sharing

Annotations may be saved with permissions which allow sharing. Hencegroups of users may access annotations or the annotation may be madepublicly visible. The permissions may be set to allow members of thespecified group to modify the annotation and either save within a newannotation set or save back to the original annotation set.

Once an annotation set has been created and saved to the server, it isconsidered a ‘template’ for manifestation on future instances of theannotated web page. Each such manifestation is an ‘annotation instance’,it is allocated a unique ID which allows communication betweenannotations instances. Such communication could, for example, be used tosupport shared activity such as chat or co-operative drawing.

Alternative Implementations

It will be recognized by those skilled in the art that there can bealternative implementations of the above described modules andassociated data structures. For example, rather than a triple whichreferences the content, one could implement a 4-tuple which includes thecontent, or even a 5-tuple which includes the content and data about thetarget page and page objects. It will however, be recognised that suchalternatives which embody the ideas of content and targets linked byassociation do not differ materially from the described implementation.

In another example of an alternative implementation, rather thanmodifying the behavior of the annotation content with MCD, such controlcould be implemented by other means, such as the use of controlannotations. It will however, be recognised that such an alternativeimplementation strategy does not differ materially from thefunctionality embodied in the described implementation described.

Program Components

The browser extension with which the user interacts to create annotationcontent and place it on a web page consists of three co-operatingmodules. These modules are the ‘core module’, the ‘semantic analyzermodule’ and the ‘annotation module.’

The ‘core module’ provides functionality that is persistent from thetime of installation of the browser extension although it may bemodified by software updates. Such modification may be to fix softwarebugs or to extend and/or reduce the functionality of the core module.The core module is written in a programming language appropriate to thebrowser (such as JavaScript) and uses browser specific APIs (such asMozilla's XUL—XML User Interface Language and XPCOM Cross PlatformComponent Object Model—for the Firefox browser) to access browserservices.

The core module provides multiple functions. It is the primary interfaceto the browser, the operating system and communications capabilities forthe other browser extension modules and delivers general purposeannotation functionality which is accessed by means of the CoreApplication Programming Interface (‘Core API’). The Core API provides asingle interface that hides the detail of the various client operatingsystems and browsers. Furthermore, dependence on the API of a particularclient operating system such as Microsoft Windows or MacOS, or of abrowser such as Microsoft Internet Explorer or Apple Safari is therebylocalized facilitating program portability.

The core module implements common user interaction policies; provideslayout management for annotations; co-ordinates communication betweenall other modules; provides annotation manifestation management; handlesevent management; allocates system resources, executes Annotation AccessProtocol (AAP) handling (see below) and manages the save and restore ofthe annotations. All user interaction to place, attach, move, resize anddelete annotations is handled by the core module. This ensuresconsistency of the program user experience. Layout management ensuresthat effective use is made of the browser display area and allows a userto re-position annotations when permitted. For example, page attachedannotations could be stacked vertically down the right or left hand sideof the browser display area and other annotations kept close to theirattached object.

The core module allocates an identifier to each placed annotation. Thisidentifier, unique within the set of annotations placed on the web page,is used by all modules to refer to the annotation when making API callsand is included in the data payload of events associated with theannotation. Other modules make requests on the core module using itsAPI. Depending on the nature of the request, the core module may respondsynchronously or asynchronously via the annotation event mechanism.

The ‘semantic analyzer module’ comprises an extensible set of ‘semanticanalyzers.’ There may be more or less such components in the module. Thepurpose of semantic analyzers is to undertake semantic analysis byanalyzing page content and structure to identify targets and generatetarget points. This is a two stage process and each stage is undertakenby a specialist sub class of semantic analyzer:

1. ‘structure analyzers’ identify candidate targets

2. ‘content analyzers’ refine or filter candidates

When an annotation is first created, it is a structure analyzer thatdetermines which page objects are viable candidates for annotation withthe selected tool. When a candidate is selected it is the contentanalyzer, with its specialist knowledge of that target type, thatenables the user to refine the target point.

When an annotation is subsequently manifested it is a structure analyzerthat uses target locating data to identify candidate targets to whichthe annotation may possibly be attached. The structure analyzer thenpasses the set of viable candidates to the appropriate content analyzerwhich uses target locating data to filter the set to find the candidatetarget object and then uses its expert knowledge to generate the targetpoint.

The structure analyzer discovers candidate targets based on:

1. the type of page object to which the annotation is to be attached.

2. the structure of pages of a particular web domain (e.g. facebookpages).

The generic structure analyzer can analyze any web page by mining theDOM to extract the set of XML element nodes which contain page contentand thereby identify candidate targets by type.

Because all web pages are collections of objects of various types,analyzers may be written which understand such objects. Knowledge of anytype may be embodied in a structure or content analyzer.

Specialist structure analyzers may be constructed with knowledge of thestructure of a specific web page; for example the page www.zzz999.com.Such analyzers use their knowledge of the web page to identifycollections of page objects as potential targets; for example, a newsfeed might comprise a <div> element enclosing several <p>, <img> andother elements. Such page object collections, forming a sub-tree of theweb page document, are given a type which identifies the collection andallows selection of an appropriate content analyzer. Such types, thatrepresent the structure and hence relationships of collections ofdocument elements, are ‘system defined types’.

Hence, it will be understood by those skilled in the art that structureanalyzers may operate on both media types and system defined types(hereinafter ‘type’ or ‘types’.)

Specialist content analyzers can embody knowledge not just of the typesof objects included on a page (directly or by reference) but of theobject's source to implement more detailed analysis of the content thanwould otherwise be possible. For example, a text content analyzer tiedto www.zzz999.com (the source URL) could use its understanding not justof the text type but also of the detailed formatting of phrases,sentences and text blocks of the source to provide targets that have asemantic meaning within the context of information related to the pagewww.zzz999.com. It could, for example, understand that a particularphrase is not just text but is a news feed. All content analyzers haveknowledge of types and (if relevant) sources and each such analyzer maybe said to ‘understand’ its target object type and (if relevant)sources.

Using its above described specialist knowledge of the candidate type thecontent analyzer takes the set of candidate targets from the structureanalyzer and refines targets down to the smallest appropriate piece ofcontent, where ‘appropriate’ is an attribute of the semantic contentanalyzer. This could be a single word or phrase in the case of thedefault text semantic content analyzer, a single pixel in the case ofthe default image semantic content analyzer or a specific frame in thecase of a video semantic content analyzer.

No matter how large or small the piece of content identified by acontent analyzer, a distinct point is required for spatial association.The content analyzer is responsible for determining an appropriatespatial location for the piece of content. This could, for example bethe center of a word or image, the top left corner of a paragraph, the‘centre of gravity’ of a phrase that runs over multiple lines, and soon. Hence the content analyzer generates the target point.

During annotation create (with a human user), the content analyser usesvisual feedback to interact with the user in refining the precision ofthe target and, hence, to generate a target point. The feedbackmechanism is either a default provided by the core module or aspecialist method implemented by the content analyzer.

The semantic analyzers play a critical role in the provision of semanticintegrity because once the target point is generated by the semanticanalyzer in the first instance the core module can calculate theassociation offsets from the annotation point and thereby specify thespatial and/or temporal relationship between the annotation and thetarget. This association is reproduced in subsequent instances of thedocument so that when the annotation is manifested in subsequentinstances of the document the user's semantic relationship is maintainedi.e. the annotation has semantic integrity.

The core module embodies a semantic analyzer factory that matches andinstantiates the appropriate semantic analyzer as required by othermodules. Semantic analyzers are loaded into memory as necessary. Thespecialist services of the semantic analyzers are provided via theSemantic Analyzer Application Programming Interface (SA API). The coremodules instantiates structure and content analyzers as appropriateduring annotation create. Annotation tools may also use the SA API tocreate additional targets and hence to implement multiple associations.

The semantic analyzer module (1) separates the logic for suchassociation from the annotation—i.e. each annotation does not have toprovide an implementation of semantic behavior, and (2) supportsextension of the semantic behavior of such associations by downloadingnew or improved semantic analyzer components.

The invention provides a set of semantic analyzers covering a range oftypes and sources. Some are specialized by content and others bystructure. This set, and hence the functionality of the program, may beextended by interacting with the server and downloading and installingfurther semantic analyzers. In like manner, semantic analyzers may beupdated to fix bugs or enhance functionality. Those skilled in the artwill recognize that new types and sources may be readily supported bywriting new semantic analyzers. Whilst a given implementation of theprogram will not support all types and sources the ready extensibilityof the described program architecture means it is, in principle,possible for the program to do so.

The ‘annotation module’ is the set of ‘annotation tools’ (hereinafter‘(tools)’) There may be more or less such components in the module.Unlike the core module, which delivers general purpose functionality, oreven the semantic analyzers which are more specialized but most of whichnonetheless embody knowledge of general applicability (e.g. a givensemantic analyzer may provide services to many tools), the tools aremore specialized. Each such tool implements a particular style ofannotation.

Each annotation tool has configuration information which includes:unique identifier, name, icon, default target specification (optional),default annotation point, default manifestation control data, targettype selector (optional), page source URL pattern (optional), pageobject URL pattern (optional). The server is responsible for allocatingthe tool identifier when the tool is registered. This configurationinformation is used to filter the applicability of the tool on thedisplayed page. The type selector specifies a set of web page objecttypes that are candidates for annotation by the tool. It may be empty,in which case any visual page object, including the page itself, is acandidate for annotation. The page URL pattern is used to determinewhether or not the tool can be applied to the current page—if the matchfails, the tool is not made available for selection. The page objectsource URL pattern is used as a filter for page objects whose content isnot contained directly within the displayed web page. If the matchfails, the page object is removed from the set of candidate targets. Ifapplication of the page object source URL pattern results in an emptyset of candidate targets, the tool is not made available for use. Itwill be recognised that there are many techniques for pattern matchingon URL strings and choice of a particular technique is not material tosuccessful implementation of the embodiment.

A tool is itself a set of components comprising the ‘icon’, the‘editor,’ the ‘media’ and the ‘helper’.

The ‘icon’ is a graphic element that represents the tool in menus, suchas the user palette, and onscreen during the annotation process. Theicon depicts an image to remind the user of tool functionality. The iconmay be created by any one of the many available graphics programs. Anopen source example of such a program is GIMP. A widely used proprietarysuch program is Adobe's Photoshop. All tools in the describedimplementation have an icon although those skilled in the art willrecognise that there are alternative means to refer to tools and thatsuch alternatives are not material to the invention. Icons are used byhuman users to interact with the program but computers accessing theprogram programmatically by means of the Annotation API do not interactwith icons.

The ‘editor’ implements the distinctive annotation capability of thetool. It creates the annotation and embodies tool specific userinterface functionality. Hence, for example, if the tool is a paintprogram the editor implements the program's characteristic ability toexecute paint operations. The editor is required only during creationand editing of an annotation. Should the user choose to modify anannotation then, assuming he has the necessary access permission, theappropriate editor is re-loaded into memory. If the media (see below)consists of a single non-revisable image (aka clip-art), then anannotation editor is not required. There is a distinction betweenediting an annotation and handling an annotation. The only program thatever edits an annotation, manipulating its content, is the editor. Inthe case of a visual annotation it may be said that the editor owns thecanvas on which the annotation is created.

The ‘media’ is a collection of pre-created media objects from which theuser can make selections to annotate web pages or will be used by theeditor to assemble the annotation. Examples of such media objects wouldbe a clip-art image of a speech balloon, a vector graphic element or aFlash movie element. Such media may be of many types and may thereforebe created in many ways. A graphic image may be created using programssuch as GIMP and Photoshop. Audio media may be a recording captured by aprogram such as Audacity's Audio Recorder. A video media object may becreated using a program such as Apple's iMovie or it may be a URIreference to media on a website that hosts videos, such aswww.youtube.com. An image may be a link to a website such as Flickr ormay be an image file downloaded from a camera or phone in a format suchas JPEG. Media objects may also comprise multiple media types eitherprecomposed into a multimedia object or provided as individual items forcomposition by the editor. Those skilled in the art will recognize thatmany programs can be used to create many types of media objects and thatsuch objects, however they may have been created, may be subsequentlyimported into and used by the program. There is a distinction betweenmedia which is revisable and media which is not. A notelet, for example,is revisable; users may type text into such an annotation. A piece ofclip-art is typically not revisable; it may be handled but not edited.Tools that exclusively use non-revisable media will typically not usethe editor. Not all tools include and use media.

The ‘helper’ component renders the annotation using non-revisable mediaor data created by the editor and is required for both creation andlater display of the annotation. It also co-ordinates with the layoutmanager component of the core module to determine the positioning of theannotation. The helper specifies constraints determining how theannotation can be moved and/or resized and redraws the annotation if itis moved or resized, invoking the editor as necessary should the move orresize require changes to annotation content. The helper may renderannotation content of two or more media elements in a sequenceimplemented by the helper. By this means the program can, for example,provide a visual animation or implement an audio playlist. Suchannotation sequencing and control is enabled via the annotation eventmechanism: the core module can provide timer events when so requested bythe helper; the core module can also send named events when it enables,disables or moves an annotation; helpers can also generate events whichmay be consumed by other annotation helpers. A special type of tool, acontrol tool, has neither visual nor non-visual content. Its helperresponds to and issues events to affect the behaviour of otherannotation helpers as described above.

The separation of functionality between editor and helper ensuresminimum browser memory space use for a user viewing annotations. Thisdivision of responsibility between ‘editor’ and ‘helper’ is:

1. editor

-   -   a. interaction with the user to create and revise the annotation        content

2. helper

-   -   a. render the annotation within the area specified through        negotiation with the core module;    -   b. specification of annotation association constraints (if any)

It is possible for all tools to directly embody expertise of types. Forexample, tools that annotate text blocks could incorporate knowledge oftext objects. The described implementation however places such commonexpertise about text objects into content analyzer(s) which areaccessible to any number of such text annotation tools. Similarly,content analyzers for other media types encapsulate expertise aboutthose media types. By this means the specialized knowledge about a giventype or source may be made available to whole families of tools. Thiseases maintenance and the development of tools.

The invention provides for a set of tools covering a range of annotationmedia types and styles. Some tools can create content which may be usedto annotate any type of page object e.g. notelets. Other tools can onlybe applied to specific content or specific web page(s) by usingspecialist knowledge of particular source URL content. For example: atext highlighter can only highlight text; an eBay auction notelet usesits specialist knowledge of eBay web pages to extract auction listingcontent to pre-populate notes for annotation of eBay auction pages.

The program includes a set of tools, suitable for annotating differenttypes of content in different styles, and which are loaded into memoryas necessary to support user annotation requirements. The inventionprovides for the development and integration of further such tools; theymay be downloaded and installed by interaction with the serverfunctionality. The ability to integrate add-on functionality as requiredby means of dynamically loaded annotation modules enables the program tosupport a broad range of annotation media and associated annotationtechniques whilst managing the memory footprint of the user's browser.Furthermore, as will be recognized by those skilled in the art, thedesign of this embodiment enables the future integration of presentlyunanticipated annotation media types and techniques by means of newtools.

The invention enables the development of novel tools which are unknownin the prior art. For example, a tool which allows a user to pull a‘query’ object onto any page on the web, ask a question about specificpage content about which he may be unclear, and to publish that query toa group or set of groups. Such groups could include a study group, or itcould be a social search engine such as www.quora.com which harnessesthe intelligence and knowledge of people with access to the web. Repliesto the query would automatically be routed back to the annotation andthe user would see them displayed as answers to his query. By this meanscollective intelligence may be brought to bear across all content on theweb.

It will be recognised by those skilled in the art that the support forannotation tools provides a framework for, and the pool of annotationtools provides a realization of, a means by which the program mayreceive annotation input data constituting annotation set content forannotating targets in web pages. It will be further recognized that thecombination of such annotation tools, the Annotation API and the CoreAPI represent a comprehensive means by which the program may receiveinput for modification of existing annotations or receive input for thecreation of new annotations.

Tools access the core module and the semantic analyzers by means of theCore API and SA API respectively. The browser extension accessesoperating system services, such as file access and communications, bymeans of the browser API thereby insulating the browser extension fromthe details of the various system call interfaces of the variousoperating systems.

Because the core module and semantic analyzers provide the functionalityrequired to support semantic integrity, the burden of providing semanticintegrity functionality when writing a new tool is eased for thedeveloper. For example, the tasks of selecting and refining targets,evaluation of association offsets, re-finding targets and displayingannotations in accordance with their targets and associations is handledby the core and semantic analyzer modules. The tool developer need onlyaccess such functionality through the Core API and SA API.

Server functionality is implemented by storing all annotation set datain a backup store such as a database along with supplementaryinformation about the annotation set in question. Examples of suchsupplementary information are: the URL of the web page associated withthe annotation set; tags describing the nature of the annotation; andaccess control permissions. The server provides a set of programs tomanage and access this data. The server allocates a unique identifier tothe stored annotation set: the Unique Set Identifier (hereinafter‘USI’). This unique resource identifier enables:

-   -   the annotation set creator to later access, download and update        the set or to delete it if required    -   annotation set sharing    -   annotation set linking

Linking annotation sets implements a linked annotation space ‘on top of’the world wide web. Hence, in much the same way that users on the worldwide web may navigate from web page to web page by clicking on displayedlinks, users may navigate from annotated web page to annotated web pageby clicking on displayed annotation link objects (a web page <anchor>element with a USI as its ‘href’ attribute value). If, for example, auser is on the page www.zzz999.com/123 and actions a USI link on thispage then the user's browser will change the displayed page fromwww.zzz999.com/123 and will instead display the web page and associatedannotation set referenced by the USI. This linked-to new page with itsannotations could be any properly addressed web page with itsannotations e.g. www.zzz999X.com/345.

FIG. 16 is a high level representation of the relationships between aUSI anchor 2210, the annotation set data 2230 it refers to and theannotated document 2250. As the USI is a resource identifier string,just like a URL or URI, it may be used as the target of a hyperlink(HTML element <a>). Thus, USI anchor 2210 is an HTML <a> elementembedded within web page object 2216 on web page 2218. The USI hyperlinkdestination is a reference 2220 to annotation set data 2230 in anannotation store 2232. The annotation set data 2230 includes a reference2340 to the annotated document 2250 held in document store 2252.

Such annotation links use the same universal resource name format as webpages but with a unique scheme name to identify the protocol forretrieving web pages and associated annotation set. The ‘AnnotationAccess Protocol’ or ‘AAP’: for example the USI string would be of theform: aap://annotation_set_id. The core module installs a handler forthe ‘aap’ scheme so that any reference to a USI is handled by theprogram. It will be recognised by those skilled in the art that adifferent scheme name may be used without compromising the architectureand that the scheme name must be registered with LANA (the InternetAssigned Numbers Authority). Traversal of an ‘annotation link’ isactioned by an annotation helper either as the result of userinteraction with the annotation content or on receipt of an annotationevent. The core module uses the USI to refresh the browser display withthe referenced web page and immediately displays the annotations fromthe annotation set.

Those skilled in the art will recognize that the functionality wherebythe program stores and manages information on the server represents the‘model’ in the Model View Controller (MVC) architecture. The annotationsmodel may be accessed by multiple controllers and the results displayedby many views. For example, annotations may be searched and retrievedand the results used in many ways, subject to access control. They may,for example, be forwarded to a user's computer to be displayed by thebrowser extension module, they may be displayed on a website (this viewdoes not require use of the browser extension), they may be madeavailable as a feed to other programs, and they may be analyzed andstatistics abstracted for later display. By this means, annotations maybe shared and commented on and may thereby become ‘social objects’, aterm of art familiar to those skilled in the art.

It will be familiar to those skilled in the art, that the term ‘server’may refer to a single computing device supporting all software requiredfor provision of the necessary functions and it may also refer to aco-operating distributed network of computing devices each of whichsupports one or more of the required functions. The topology of theserver implementation is not material to the feasibility of theinvention.

It will further be recognised by those skilled in the art that whilstthe program implementation describes program modules and componentswhich are not only logical functional blocks but are in some casesdiscrete modules which may only be downloaded and integrated as neededit is perfectly possible to build alternative implementations whichembody the described functionality but which do so in a more tightlyintegrated manner. Such integration can extend from combining semanticanalyzers and helpers up to and including a version of the program inwhich all the described functionality is embodied in one huge programand in which case extending the program is done by updates to saidmonolithic program. The implementation described in detail here hasadvantages in terms of modularity, ease of maintenance, browser memorymanagement and ease of extension and updates but such alternativeimplementations, employing the ideas described, whilst not novel ordifferent in terms of core functionality, are perfectly feasible.

Program Operation

It will be described below how to install the browser extension, how toenable it, and how it is used and configured. It will further beexplained how it operates and how its component modules cooperate tocreate, edit and attach an annotation set to a web page and how thebrowser extension cooperates with the server to save, retrieve andrestore, and share annotation sets with semantic integrity.

(It will be appreciated by those skilled in the art that the descriptionbelow focuses on the interaction between a human user and the program.It will further be appreciated that much such interaction is notrequired when a computer program user accesses the program and theannotation data structures by means of the Annotation API.)

At this point it is helpful to elaborate further on the notion ofsemantic integrity and the related notion of semantic analysis. Semanticintegrity is derived from the spatial and/or temporal relationship ofthe annotation manifestation content to other page objects. Specificallyfrom the relationship between the annotation content and the target. Theprogram is designed to capture the semantic relationship which definesthe location of the annotation in space and/or time so that it many bereproduced (and the user's intent thereby reproduced) in subsequentinstances of the manifestation of the page with the annotation. To thisend the program determines the target and hence the association betweenthe content and target by content and structure analysis. This processis described hereinafter as ‘semantic analysis’ and program moduleswhich perform such analysis are the above described ‘semanticanalyzers.’

Semantic analysis as described herein is the process whereby, when auser creates or edits an annotation and interactively specifies theannotation's target, they thereby express their intended semanticrelationship between the annotation content and the underlying web page.The program analyses this relationship between manifested annotationcontent and target to enable the program to thereby capture the user'sintended semantic relationship and reproduce this relationship insubsequent instances of the web page, thereby enabling the manifestationof subsequent annotations with semantic integrity. The results ofsemantic analysis are encapsulated within the Target Locating Data andstored as part of the Annotation Data.

Semantic analysis in the context of the program always involves theanalysis of content to determine the association. It is therebydistinguished from any alternative indirect method that locates anelement within the structure of a document. For example by means ofcapturing an element's xpath on the DOM tree. Such an approach uses therelationship between elements in metadata as a proxy for the actualrelationship between manifested content. As those skilled in the artwill appreciate, this approach may be successful for some pages butbecause the association is derived indirectly—the approach does notanalyze the relationship between annotation content and target butrather between elements of metadata—it is susceptible to a class oferrors arising from modifications to the page structure or modificationsto object content and furthermore, it cannot address temporalassociation at all. Such an indirect approach, which undertakes nosemantic analysis, can at best find the same node on the DOM tree asreturned by evaluation of the xpath expression, and hence cannot be saidto be providing semantic integrity in the sense described herein.

It will be recognised by those skilled in the art that whilst it ispossible to take a copy of a web page, annotate that copy and then onrequest re-manifest the copied page with its annotation, such animplementation cannot address the challenge of web pages themselveschanging. The annotated page displayed in the later instance is not thenecessarily the page which would be displayed at the URL of the originalinstance of the page and indeed this page may differ radically in layoutand content. Hence such an implementation cannot reliably annotatechanging web pages and does not realize semantic analysis or semanticintegrity as described herein.

Setup

The browser extension core module may be downloaded over the interne andonce installed it adds a ‘program icon’ to the browser's ‘Tools’ or‘Options’ menu bar. Those skilled in the art will know that the toolbaris known by different names on different web browsers but that suchdifferent naming conventions are not material to the operation of theprogram.

Selecting the program icon gives the user access to the program menuwhich offers top-level configuration options allowing him to: a) enableor disable the extension, b) provide user authentication information(login to the server) and c) access authorized server hosted supportresources e.g. profile pages.

(a) Enable Program

When the user selects the program icon and selects ‘enable program’ fromthe program menu the core module undertakes the following actions to setup the program in readiness for the user to annotate: if the user is notalready logged into the system the user logs in to authenticate himselfand thereby gain access to program controlled resources. If the userdoes not already have an account on the server, he is prompted to createan account. Those skilled in the art will recognise that a variety oftechniques are commonly used for creating and validating user accountsand for authenticating user logins and the technique employed is notmaterial to the nature of the embodiment. The server allocates a uniqueidentifier to the user's authenticated annotation session. Thisidentifier is used whenever the core module communicates with the serverand for inter-program communication. The core module then displays theannotation icon in the browser display area. This annotation iconprovides access to the available annotation tools, a list of annotationsets that have been saved for the current page and for which the userhas access and also to any annotations created by the user on thecurrent page. The annotation program is now enabled and awaiting furtheruser input. The user may disable the program at any time by selecting‘disable program’ from the program menu. When disabled, the program nolonger supports any annotation interaction nor does it communicate withthe server.

The program menu also implements a ‘configure’ item which, whenselected, provides access to the program configuration dialog. Programconfiguration data includes: user login name and (optionally) password;whether the program should auto-login when enabled; whether the programshould auto-enable on browser startup; whether annotations should beauto-saved when added or modified; whether the user should be promptedwhen leaving a web page on which there are unsaved annotations.

The program menu also provide access to shared annotation sessions. Ashared annotation session allows multiple users to co-operate in thecreation of and interaction with annotations. Hence, for example, usersTom, Dick and Harry could view the same displayed web page with itsassociated manifested annotation set and could further cooperate increating, editing or deleting annotations. This group of users may thensave such an annotation set for later retrieval and restoration. At thispoint the group may jointly determine what the access controlpermissions are for such an object that has been cooperatively created.

Creating a shared annotation session causes the server to create aunique identifier for the session (hereinafter the ‘share ID’) andallocate a data set on the server to track session members. The share IDcomprises a human readable string and it may be sent via any means tocollaborators. The program menu also offers the ability to join activesessions using the share ID. Additional security for joining sharedsessions may be implemented to avoid access by uninvited participantsand those skilled in the art will recognise that there are many suchauthentication mechanisms and choice of any mechanism is not material tothe validity of the embodiment.

The core module enables annotation auto-save when participating in ashared session. Annotations are saved to the server as soon as they havebeen attached. Annotation synchronisation is effected using messagessent from the server to all session participants. Annotation events arealso distributed among session participants. As will be appreciated bythose skilled in the art, such shared annotation may be implemented bysuch means as publish and subscribe messaging or a synchronous sharedwhiteboard but the selected mechanism is not material to the validity ofthe invention.

(b) Use Program

When using the browser extension the user may browse from page to page,annotate displayed pages and save such annotations. We describe thisprocess below.

When enabled, the core module creates an annotation set container(hereinafter ‘container’) corresponding to the currently displayed webpage. As the user annotates the page this container will be updated withinformation about the annotation set. Each tool creates an annotationdata object (hereinafter ‘annotation object’) containing annotationcontent and related annotation data and the container supports objectsgenerated by all tools.

FIG. 9 is a flowchart depicting a high level view of the program'srestore process. Whenever the user displays a web page, e.g. by browsingto a new URL, or by clicking on a hyperlink in the currently displayedweb page a page load event 905 is generated. This event is detected bythe page load event handler 910, a component of the core module and thepage load event handler 910 makes a call 915 to the annotation server920 requesting (a) the user's current program configuration and (b)details of any annotation sets saved for the loaded web page which thecurrent user has permission to access. The annotation server 920transmits a response 925 with the requested configuration data and dataabout annotation sets with any associated community information on thecurrent page that the current user has permission to access. Communityinformation could include, for example, data as to which annotation setswere created by friends, which are the most recent, or which are themost popular in the community. Receipt of the response 925 initiates aprocess: the initial step 930 of this process determines if there areany annotations for the page. If there are no annotations the processterminates and the program awaits further input at 935. If there areannotations then the program displays a list of the annotations with anyassociated community information 940. Step 945 determines if the userhas selected an annotation set for display. If the user has not selecteda set then the process terminates and the program awaits further inputat 935. If the user has selected a set then the program generates anannotation set data request at step 950 and transmits the request 945 tothe annotation server 920. The annotation server 920 transmits itsresponse 950 containing the requested annotation set data. The programthen manifests the annotation set on the current web page at step 960and then awaits further input.

If response 925 above from the server indicates that the user's programconfiguration has changed, e.g. through acquisition of additional toolsor because of bug fixes or other updates to a program module, the coremodule makes further calls to the server to request the new or updatedcomponents. The server response is handled asynchronously to avoiddelaying user interaction. By this means, for example, successfullydownloaded new tools may be presented to the user on his next programmenu access.

By means of the above described process, if there are saved annotationsets to which the user has permission to access the program can listthem for the user and, if requested, display them. We now describe thecreation of a new annotation set.

(c) Annotate

Whilst the implementation described below makes use of discreteactivities cooperating to perform the annotation, it is perfectlypossible to combine all these activities into one monolithic module, orotherwise distribute the functionality described into activities with adifferent division of labor than those described here, but those skilledin the art will recognise such a redistribution of the describedfunctionality does not differ materially from the describedimplementation.

The process of creating an annotation with which to annotate a web pagecomprises five discrete activities: create, edit, handle, attach and(optionally) delete. Tool selection generates a new unique annotationobject with an initial target as determined from any default targetspecification in the tool configuration information. As the editor isused to create annotation content, data about that content will begenerated and will further populate the data object.

The full cycle of creating annotations prior to saving includes at leastcreate and attach, which determines the target (the default may or maynot be overridden by the user). The annotation association data isinitialized by the core module using constraints provided by the helper.The user may change the annotation association data as required bymoving the annotation in which case the core module recalculates theassociation data. Once an annotation has been created, attached andpossibly handled other annotations may be created by repeating theprocess thereby creating a multi-member annotation set. Annotations andthe annotation set may be deleted at any time by the user. The layout onthe page of multiple annotation sets is determined in the first instanceby the user but is subject to policies managed by the ‘layout manager’,an element of the core module. Create is always the first action anddelete always the last; edit, move and attach may be invoked at anystage during the process. We shall now discuss the steps of annotationcreation in more detail.

(c)(1) Annotation Create

When the user selects the annotation icon in the browser display areathe core module detects the event and displays the annotation menu. Apalette of available annotation tools is either contained within oraccessed from the annotation menu. The annotation icon and user palettemay, for example, be positioned at the bottom border of the screen, butit will be appreciated by those skilled in the art that neither thelocation of said icon, annotation menu and user palette nor theirdisplay criteria, nor their ‘look and feel’ are critical to thefeasibility of the invention. The palette displays a set of iconsrepresenting the tools available for the user to annotate the currentlyloaded web page. Such icons may be displayed in accordance with variouscriteria. For example, favorites at the top level and additional toolsvia sub menus. The favorites collection could be determined by eitheruser choice, recent use or frequency of use. If a tool cannot be appliedto the current page—e.g. if no page objects match the tool's typeselector or the page URL does not match the tool's page source URL—theicon is grayed out and the icon cannot be selected.

The user initiates a tool by selecting its annotation icon from the userpalette. He may, for example, use a computer mouse device to drag theicon from the palette onto the browser's display area or he may click toselect the tool and then click on the browser display area to place theannotation. When the user selects a tool the core module detects theevent and instantiates the tool.

By whatever means, the annotation is placed and the helper is invoked toprovide an initial annotation content, which may be blank, plus:

-   -   initial distance of annotation point from and position relative        to the target    -   optional maximum and minimum distance of annotation point from        the target    -   whether the annotation may be moved relative to the target    -   whether the annotation can be resized

As noted above, the invention enables annotations to be attached to (a)the whole page (b) a page object, (c) a collection of related pageobjects (d) a region of a page object (e) a region which spans acollection of related web page objects or (f) a point within a pageobject. From the moment of its creation an annotation is always attachedto one of these targets. The program provides a default initial targetappropriate to the type of the object being annotated. This initialtarget may be changed or its precision may be refined by the user byinteraction with the core module.

To refine the annotation content the user interacts with the editor. Theeditor controls the ‘annotation space’. The nature of this space willvary from tool to tool. For example, for annotations with a visualrepresentation the editor works on a virtual canvas which overlays aportion of the display; for an audio annotation the editor works on avisualized audio timeline; for a video annotation the editor works on acanvas and a timeline and for a control annotation, a control script isused to express the actions to be taken on receipt of events.

Whereas generic ‘annotation create’ interaction is handled by the userinterface component of the core module, thereby enforcing maximalconsistency of user experience, the tool specific interaction generatingcontent in the annotation space is handled by the tool itself and mayvary a great deal from tool to tool. The process of draggingnon-revisable clip art onto the display area is very different from therevisable process of free-form painting; consequently, the editor andassociated user interface of these programs will be different. Clip art,for example, is naturally placed using a drag/drop process, whereasfree-form painting is more naturally associated with aselect/move/select process (e.g. click mouse button to select drawingtool, move mouse to where user wishes to start drawing, depress mousebutton and ‘draw’ by moving the mouse and finally releasing the mousebutton to end the drawing path).

Whereas the above description focuses on the user experience, below wedescribe how the browser extension modules cooperate during the process.

When the user initiates tool use by, for example, dragging the tool icononto the display area, it may be that the tool can annotate every typeof page object. Alternatively however it may be that its applicabilityis restricted to a subset of the displayed objects. In either case thecore module determines which objects on the displayed page are eligibletargets for the content created by the tool by the following process—asoftware factory in the core module selects and invokes a structureanalyzer in accordance with the following priority:

1. the structure analyzer specified in the tool configuration of theselected tool

2. a structure analyzer with URL match pattern; if there are multiplematches, then the structure analyzer with the longest match is selected

3. the default structure analyzer

If, for example, the type selector in the annotation tool configurationspecifies a type or types, hence indicating that the selected tool mayonly be applied to content of the specified type(s), then the selectedstructure analyzer uses the Core API to explore the web page structureand find all candidate web page objects—i.e. those containing contentwhich matches the type selector specified in the annotation toolconfiguration. In such a case, where the tool's application isrestricted, the core module provides visual feedback to the user as towhich page objects are valid targets. The type of visual feedback ismanaged by the core module and may be, for example, a border around theobject's outline, by altering the object's background color or bydimming inapplicable objects. Hence, if a tool such as a texthighlighter (which can only operate on text objects) is selected thenfeedback will make it clear which page objects are text blocks and hencevalid targets.

Annotation Handle

Once created, the annotation may be handled by the core module. Theannotation may, for example, be moved or resized, its temporalrelationship to the target may be edited, its position within thelayering of annotations may be altered, or the use of events to controlmanifestation may be managed. In the case of visual annotations, suchoperations are typically made available by user gestures on the borderof the content—directly in the case of moving or resizing the annotationand via a core module implemented dialog for temporal and eventrelationships. Access to annotations (visual and non-visual) isavailable via the annotation menu. Some aspects of the MCD may bemanipulated by the user directly from the annotation menu, these includestart delay, manifestation time, blink, fade in/out. Other aspects (e.g.attribute exposure and event handling) are better manipulated by thehelper. The annotation helper may install additional menu items toprovide simple user controls for handling these more complex aspects ofthe MCD. The core module provides menu and dialog management facilitiesto support the annotation helper with creation of such user controls.

The core module places a visual border around the annotation space andthis border contains a number of special regions for directly resizingand moving the annotation plus a region that provides access to a coremodule menu or dialog for changing an annotation. Those skilled in theart will further recognize that there are many user interface techniqueswhereby the user may interact to change such information but that suchvariance is not material to the invention.

(c)(3) Annotation Attach

As stated above, all annotations, created by the user interactiondescribed above, have an initial target which may be modified by userinteraction.

Where an annotation may be attached to a page object target or to thewhole page, the program must provided a mechanism for the user to selectthe page as target. Those skilled in the art will recognise that thereare many alternative ways of allowing the user to signal that it is hisintent that the target be the whole page. An example implementationwould be to provide a ‘page icon’ on the toolbar onto which the userdrops the notelet. Another example is the use of a ‘modifier’ key that,when depressed during the user's mouse gesture, modifies the attachmentbehavior to make the page the only available target.

An example of an annotation which may only be attached to the whole pageis one for which the tool's type selector specifies a target type thatdoes not have a visual representation (e.g. audio). In this case thecore module makes the whole page the only candidate target object anddropping the annotation icon anywhere on the displayed web page resultsin the display of a list of all page objects matching the target typeand discovered by the structure analyzer. There are multiple possibleways to display such a list e.g. via a pop-up dialog, or via theannotation menu accessed from the annotation icon. The user selects oneof the page objects to annotate. In the case where the annotation itselfis non-visual, there is no spatial annotation point or spatial offset.In the case of a visual annotation, the default semantic contentanalyzer sets the target as the whole web page and uses the origin (0,0)as the target point. The layout manager modifies the spatial offset toposition all such annotations according to a policy, either default orselected by the user. Examples of such a layout policy are: stackedvertically down the left hand side of the page; cascaded starting fromthe top left corner of the page.

The description below covers the case of a visual annotation beingplaced onto a visual page object. The core module instantiates a contentanalyzer appropriate to the page object as soon as the annotation hasbeen placed during creation and the initial target is set to a defaultfor that type of page object. If the user does not elect to refine thetarget, this content analyzer is retained and it will capture sufficientinformation from the whole page or page object to ensure that theannotation may be later restored.

To attach an annotation to a specific location within a page object e.g.a block of pixels within an image, the user elects to refine the targetwithin the current page object. In this case the content analyzer isinvoked again to support the refinement process. During targetrefinement, the user may choose to move the target to a different pageobject—another from the set of candidate page objects found by thestructure analyzer. In this case, the core module discards the originalcontent analyzer then selects and instantiates a new content analyzerappropriate to the page object chosen by the user. Those skilled in theart will recognise that there are many methods for invoking specialistbehaviour on a graphical representation and selection of a particularuser interaction method is not material to the validity of theinvention. For example:

i. the core module may decorate the annotation with visual borders thatsupport mouse driven interaction either via gestures on specificportions of the border or via a menu accessed via a click on the border

ii. the target may be visually represented (e.g. using a crosshair icon)and moving the icon invokes content analyzer selection.

Once the user so elects, the appropriate content analyzer isinvoked—i.e. the content analyzer which ‘understands’ the page objectwithin which the target will be located. It is the content analyzer'sspecialist knowledge of the page object that enables it to provide themore detailed information about the object that makes selecting a targetat any degree of resolution possible. At this point the user enters intoa further interaction which enables him to drill down into the pageobject and select whichever subset of the object he wishes to serve asthe target.

The core module provides default interaction methods for selection oftargets on text and image page objects. Target selection is made in atext block using the mouse cursor to ‘sweep out’ the requiredcharacters, words or phrases which are highlighted to show theselection. Target selection is made in an image by using the mousecursor to reposition the edges of the ‘rubber band’ outline which isinitially co-incident with the image boundaries. Those skilled into theart will recognise that other techniques are possible. A specificcontent analyzer may provide a bespoke interaction method—for example,to select a point on the time line for an audio or video page object.Those skilled in the art will recognise that implementation of such abehaviour override mechanism is a well understood method of art.

Page object data not contained within the web page itself but ratherincluded by reference requires the support of a special component(hereinafter ‘manifestation component’) to manifest the content. Mostweb browsers have built-in such manifestation components for commonmedia types such as bitmap images and audio streams. Other media typesrequire the installation of a software component to read the data andmanifest the content in a form appropriate for the user—e.g., bydirecting audio to the operating system sound system and by rendering ananimation corresponding to a video stream. Some such components requireinstallation but at the time of writing are extremely widespread e.g.Flash players. Those skilled in the art will recognise a component ofthis kind as a browser plug-in which consumes the data (hereinafter‘data consumer’) and that it may be downloaded and installed byinteraction with the browser Setup or Option menu.

Content analyzers for such page object data that is not contained withinthe web page may choose to substitute their own data consumer in orderto generate annotation events derived from temporal aspects of the data.This data consumer is active during annotation creation to analyze thedata and later during the manifestation of the annotation to generatetarget timeline related events. For example, the target on an audio orvideo data stream could be defined as when a certain musical phrase, orscene in the case of video, appears. Those skilled in the art willrecognise that several techniques are possible for implementation andlocation of such a data consumer. For example an intermediate datafilter could be inserted into the data stream via URI re-write. Streamedobject data is referenced on its source using a URI (Uniform ResourceIdentifier) and this URI is encoded in the page object element. Thecontent analyzer modifies the page object DOM element to substitute theURI of its own player output and uses the original URI to access thestreamed data and generate annotation events derived from analysis ofthe data. The data consumer copies input to output so allowing the pageembedded player to work as it would do on the unannotated page. (Such adata consumer could be installed on the user's computer or could beimplemented as a server function.)

Browser plug-ins may have a visualisation and user interaction mechanismwhich is specific to the particular plug-in. An alternative (orenhancement) mechanism to the data consumer substitution for contentanalysis is to interpret the visualisation of the data. Examples of suchvisualisation are: a) an audio player which displays the timeline as amoving bar with values above the bar showing the current and total playtime and b) a video player which displays the scene name and/or numberplus current and total play time. Those skilled in the art willrecognise that there are several mature techniques for analysis of suchan animated image in order to extract contextual information as data andthat the architecture of the described embodiment supports selection ofa technique appropriate to the particular plug-in. As such datavisualisation is logically outside the browser, the content analyzerrequires access to the picture elements (hereinafter ‘pixels’)corresponding to the computer screen location of the plug-in within thebrowser display area. This is possible because all operating systemsprovide a method for accessing the display buffer and the core modulecan implement an abstraction of the operating system native API. By thismeans a content analyzer may request, via the Core API, access to theoperating system's display buffer so it may, for example, take regularsnapshots of the pixels in order to perform image analysis. By thismeans semantic analysers may identify targets by direct analysis of themanifestation of the web page content.

Some styles of annotation, such as the drawing of a line or arrow, maydemand the definition of more than one target, one for the start of theline and another for the end of the line. In this case the first targetdefines the start, the annotation helper then requests a new target bycalling the core module to track the mouse again, another contentanalyzer is created, and the second target is analyzed as was the first.

Once the target is determined it serves as an anchor about which theuser may move the annotation at will, subject to the associationconstraints.

To implement the above described functionality the program needs to knowthe location of the displayed cursor. Those skilled in the art willrecognise that DOM events, some specific to particular browsers, areavailable for tracking mouse movement. By this means the core module candetermine which web page object is under the mouse cursor using the DOMevent mechanism. Objects within the DOM representation of the web pagesupport an event mechanism allowing interested program components to benotified about operations on the object. By setting up handlers tomonitor such events and analyzing the resultant information stream thecore module determines which page element type the mouse cursor is overand where the mouse cursor is in relation to the page elementboundaries. As the cursor moves, the core module constantly updates thisinformation, recognizing, for example, when the cursor crosses pageobject boundaries and is over a new element. If the mouse cursor is notcurrently over a page object of interest, then events associated withthe page body can be used to determine the nearest page object ofinterest. If the mouse cursor is not over or near a valid candidate webpage object when the user terminates the attachment process, theannotation operation can be aborted. Those skilled in the art willrecognise that whilst the above description describes a mode of userinteraction with mouse and cursor, there are alternative models of userinteraction, such as touchpad and gestures, which may involve differenthardware, and that these alternatives also generate event streams thatmay be used to implement the program's functionality.

At this stage of the process the user has created the annotation contentand by means of his interaction with the program and calculations madeby the program, the program has determined all required relatedannotation data. The user has, for example, determined the target (page,page object, region or point in the case of spatial, and instant orinstants on the timeline for temporal) and the program has thereforegenerated the target locating data; has resized or moved the content asnecessary to settle the display location (and the program has derivedthe spatial and/or temporal associations and has interacted with theprogram to specify any event driven behavior.

(c)(4) Annotation Edit

Annotation content may be derived from the content of the page objectbeing annotated. An example of this is text highlighting where theregion selected on the annotated page object is decorated to make itstand out from surrounding text. Another example is a notelet annotationwhich is pre-populated with some or all of the attached region text. Theeditor uses the Core API to request the content of the attached region.

The editor may need to display additional user interaction elements—e.g.a paint tool palette. Those skilled in the art will recognise that thereare many techniques for the implementation of such user dialogs andassociating the manifestation of such dialog with user interactionwithin the annotation space. Those skilled in the art will furtherrecognise that choice of such technique is not material to the validityof the described embodiment.

(c)(5) Annotation Delete

At any point prior to saving the annotation it may be deleted. This maybe done either by explicitly accessing a command on the program menu,via the core module maintained annotation border or by abandoning theannotation by navigating away from the displayed page without confirmingthat the annotation set be saved (assuming that annotation ‘auto-save’has not been enabled via the program menu). Furthermore, savedannotation sets may be subsequently deleted.

(d) Annotation Save

The user may now either save the annotation as-is or select another tooland repeat the above process to further annotate the page thereby addinganother member to the annotation set. Each member of the set that has avisual representation may be thought of as a layer over the content.Similarly non visual representations may be layered, audio may forexample by layered on audio. Annotations may therefore overlayannotations. The layout manager implements policies which determine howannotations overlay each other and the user may modify this policy byediting the annotation set data—e.g. via a menu accessed via the coremodule maintained annotation border. In the described embodiment thedata for all annotations are held in memory and any annotation may berepositioned by selecting its representation on the browser display. Theorder of such overlays (front to back) may be changed by the user, as ingraphics drawing programs such as GIMP, in a manner familiar to thoseskilled in the art. In such cases the appropriate helper and contentanalyzer are automatically invoked and a selected annotation may bemoved to a new location or otherwise modified as permitted by the tool.Such further annotation steps may be repeated, until the user issatisfied with his work and elects to save the annotation set created onthe page.

Once the user elects to ‘save’ by selecting an icon or menu item in theuser interface, the stored state of the set of annotation objects may besaved to server storage. Should the user browse to a different web page,all data corresponding to the annotations on the previously displayedweb page is deleted from memory. The user may choose to save theannotations before displaying a new web page. If the user does not save,the core module displays a dialogue pop-up window to ask him if hewishes to save his annotation. Annotations within a shared session areautomatically saved by the core module as soon as the annotation hasbeen attached and whenever the editor has changed the annotationcontent.

When the user saves the annotation set he does so with associatedinformation that enables the program to search, browse, and share it.Examples of such information are permissions, which may be more or lesspermissive and are used by the server to control access to annotations,and tags which enable users to search and browse annotations.

The save is implemented by the core module and results in serializedcopies of each annotation object state being transmitted to the serverwhich allocates a unique ID to the annotation set. Each annotationtriple has an associated structure and content analyzer. The internalstate of these analyzer objects encapsulate the target locating datawhich comprises sufficient data for finding and restoring the target.FIG. 10 is a diagram depicting the annotation save process 1000. Thecore module 1020 receives an annotation set save request 1010 and sends‘save state’ events (a) 1050 to the semantic analyser module 1030, and(b) 1070 to the annotation module 1040. The semantic analyser module1030 serializes the state of all structure and content analyzers andtransmits 1060 that serialized data back to the core module 1020. Theannotation module serializes the state of all helpers and transmits 1080that serialized data back to the core module 1020. The core module 1020marshalls this annotation content data for all members of the annotationset, with all associated target locating data and association data,together with the annotation set attribute data, comprising such data asuser credentials, web page URL, permissions data and tag data(annotation set attributes are described in more detail below), to theserialized component data and transmits the entire data payload to theannotation server 1090 as the annotation save operation 1085. Theannotation server 1090 responds 1095 by returning an annotation set IDto the core module.

Those skilled in the art will recognise that there are many techniquesfor associating such state data, as embodied within the target locatingdata, with a module or collection of related executable code thatmanipulates the data; that the described encapsulation within an objectis just one such technique; and that the choice of technique is notmaterial to the feasibility of the invention. Those skilled in the artwill further recognise that the term ‘serialized’ refers to a method offreezing the state of an object and converting it into a sequence ofbits so that it may be stored in a file or memory buffer and laterrestored to create a semantically identical clone of the originalobject. They will further recognise that the method for creation of suchserialized copies depends on the programming language employed and is anaccepted technique of art. The precise method used for serialization ofthe annotation components is not material to the feasibility of theinvention.

Although the serialized annotation set data may be opaque to the server,those skilled in the art will recognise ‘serialized’ data is data whichmay appear opaque but which may be parsed and its underlying datareconstituted. The ‘annotation set’ may be represented, for example, byan XML document which encapsulates the attributes and serialized objectdata for each annotation member. It would be perfectly possible toimplement alternative embodiments in which the server does understandthe structure of the annotation set data.

In the case of multiple annotations on a single page the describedembodiment may elect to automatically save each annotation member on apage in turn as they are created. By this means, an incomplete set maybe recovered and completed should the user's browser crash or networkconnection fail during a sequence of annotations.

The implementation gives the user the option of specifying tags to beassociated with the annotation set. As is familiar to those skilled inthe art, tags may be described as text used to categorize objects, in afree form manner i.e. there is not necessarily a pre-defined ontology.Those skilled in the art will further recognise that there are manymeans of associating information with the annotation to enable futuresearching and browsing of such annotations and that the chosen scheme isnot material to the feasibility of the invention.

Annotation sets are saved to the server by storing the data for eachannotation in the form of serialized opaque annotation object(s) plusany information required to reproduce the annotation set faithfully asmanifested at the time of creation, plus information to control accessto the set. Such information will therefore include, for example:

-   -   unique annotation set ID    -   annotation set name and/or title    -   authenticated ID of logged-in user    -   URL of underlying annotated document    -   annotation ordering, layering and layout information    -   permissions and sharing data    -   annotation set description    -   tags

Such data is referred to as the ‘annotation set attributes’.

It will be appreciated by those skilled in the art that the abovedescribed process for storing an annotation set represents an annotationstorage means for storing annotations.

(e) Annotation Retrieve Restore

An important consequence of storing the full annotation data for allannotations is that annotations may be faithfully recreated. The contentwill be manifested as created, the spatial relationship to the targetwill be restored (determined by the spatial offset subject to spatialconstraints and layout policies), the temporal relationship to theobject timeline will be observed (determined by the temporal offsetsubject to temporal constraints) and the manifestation will behave inaccordance with the MCD. This ability of the described embodiment tofaithfully restore annotations in this manner is the property ofsemantic integrity.

Hence, for example, because visualized annotations have spatial andtemporal offsets and event specifications even if, for whatever reason,targets move around on the screen, or vanish from and then reappear onthe screen, a displayed annotation retains a defined spatial andtemporal relationship to its target and will track the target moving onthe screen, vanish with the target and reappear with it.

For the purpose of simplifying the description below, we assume that theannotation being restored has a single target. For annotations withmultiple targets, the process is repeated for each target. The processesdescribed below are presented for the purpose of illustrating theworkflow for restoring a target. They do not represent the only possiblemechanism nor are they complete. Those skilled in the art will recognisewhere additional logic is required in order to produce completestructure and content analyzers for support of generic target handling.Those skilled in the art will further recognise how to use the skeletondescriptions below to build more advanced analyzers using thearchitecture of the program described herein.

When enabled, the extension module accesses the browser API to requestnotification whenever a web page is loaded and the program manifestsrelated annotation sets as requested and using a process as describedabove. Community generated information returned by the annotation serverresults from community accesses to and comments on the annotation setand includes, but is not limited to: free-form commentary on theannotation set; number of views of the annotation set; results of votingon the annotation set; messages from a community review process.

Browsing to a page in this way could come about due to simple browsingactivity or the result of a search of annotations. In the latter case,the search results will comprise a collection of USIs (Unique SetIdentifiers) which, when accessed by an enabled program, will result inthe display of the web page and associated annotation set. The web pageand associated annotation set will also be displayed if the user clickson a hyperlink whose target is a USI. In this case, the core modulefirst loads the web page and then manifests the annotation set.

In the case of browsing to a normal web page with saved annotation setswhich the user has permission to access the server responds with a listof such annotations. Access is determined by user profile data stored onthe server and the logged-in user's access may, for example, berestricted by:

-   -   annotation type—the user may not have the right to view all        types    -   permissions—the creator of the annotation may have set        permissions that exclude the current user

Data in the list returned from the server comprises:

-   -   number and type of annotation sets the user has permission to        access    -   name of authenticated user who created and saved the annotation        set    -   date and time annotation was saved or last updated    -   tags providing information with respect to the nature of the        annotation    -   community related information

If the number of accessible annotation sets is greater than zero theextension module uses its icon in the browser display area to indicatethat the current web page has been annotated. Clicking on the icon givesaccess to a display of information about the annotation sets on thecurrent page accessible to the user.

The core module may use one of a number of techniques for providing thisinformation—for example, displaying a number representing the number ofannotations, changing shape and/or color, animation in combination withan audible signal or any combination thereof. Furthermore, anyassociated community related information may be displayed immediatelywithin the list or it may be hidden from view until requested by theuser and it may be organised in a number of ways, e.g. by time ofcreation or according to the user's community groups. Those skilled inthe art will recognise that such incorporation of community generatedinformation into the annotation set is essential to the sharing ofannotations and that the described mechanism is extensible to support ofa wide variety of such community information.

This list may be ordered in a variety of ways e.g. most recent, friendsfirst, favorite type. The user may browse the list of availableannotations or search for annotations by tag. If the user selects anannotation for display, the extension module retrieves the serializedannotation set data from the server. The data is de-serialized torecreate the annotation components (content, triples, helper andsemantic analyzer objects). The process for de-serialization andrecreation of the objects depends on the programming language used forimplementation of the objects and those skilled in the art willrecognise that there are several techniques for such de-serializationand recreation and that the precise technique used is not material tothe feasibility of the invention.

FIG. 11 is a high level overview of how an annotation set is restored.It shows the data flow between the annotation server 1090, the coremodule 1020 and semantic analyzers 1030 for each annotation triple whena previously saved annotation is re-manifested.

The core module 1020 receives an annotation set restore request 1110 andtransmits an annotation set retrieve request 1115 to the annotationserver 1090. The annotation server 1090 uses the annotation set ID toretrieve the annotation set data and transmits the data back 1120 to thecore module 1020. The next step is for the core module 1020 to initiatethe deserialization and reinstantiation of serialized annotationcomponents.

The core module 1020 sends requests 1125 and 1126 to the semanticanalyser factory 1030 to restore the structure analysers 1141 andcontent analyser 1142. Request 1125 contains the deserialized structureanalyser state and request 1126 contains the deserialized contentanalyser state. The core module 1020 then sends a request 1127 to thestructure analyser 1141 to use its restored target locating data toidentify target candidates in the web page document. The structureanalyser responds 1128 with a set of candidate target document objectreferences 1131. The core module 1020 then sends a request 1129 to thecontent analyser 1142, the request contains the set of candidate targetdocument object references 1131 returned by the structure analyser 1141in response 1128. The content analyser 1142 uses its restored targetlocating data to filter each of the candidate target document objectreferences 1131 to identify the actual target. The content analyser 1142responds 1130 by returning the document object reference 1132 of theidentified target. If the content analyser 1142 fails to find a matchwithin the set of candidate target document object references 1121 usingits restored target locating data the document object reference 1132 isnull. In this case, the core module 1020 degrades gracefully bynotifying the user that the restored annotation could not be placed. Inthe case that the document object reference 1132 is not null the coremodule 1020 instantiates 1133 the annotation helper object using itsdeserialized state. The core module 1020 then sends a request to therestored helper 1143 to retrieve the restored annotation. The restoredhelper 1143 responds 1135 with a data set 1136 containing the annotationcontent and constraints for annotation association. The core module 1020then creates the annotation triple data and manifests the annotationcontent in association with the identified target.

By this means the annotation set members are manifested and the coremodule has access to the annotation set attribute data described above.

The restoration workflow deserializes and reconstitutes the structureand content analyzer objects for each annotation triple of eachannotation set member and also deserializes and reconstitutes theannotation helper object. These objects are restored to the same statethat existed when the object was serialized for saving the annotation.As each semantic object was responsible for ensuring that targetlocating data was saved to the object state, this data is also restoredand is used to restore the target. These objects are used to find thetarget within the new instance of the web page, to re-calculate thetarget point and, hence, to use the association data to locate andmanifest the restored annotation.

The process whereby the reconstituted semantic analyzers find candidatesfor the target and progressively filters these candidates to identifythe target is described below. The restored target locating datacontains information about the type of target and sufficient contextinformation to enable a) the reconstituted structure analyzer to find aset of candidate targets on the new instance of the web page and b) thereconstituted content analyser to filter the set of candidate targetobjects to identity the actual target and thence to regenerate thetarget point. The candidate targets are page objects which match thetarget type, which may be: the web page itself; simple types such astext or image; a more complex type such as a video stream; or a systemdefined type such as a collection of page objects.

The first step in the annotation restore process is performed by thereconstituted structure analyzer which scans the web page employing thesame algorithm as was used during annotation create, but this timelooking for page objects which match the single type specified in therestored target locating data. For example, to find the correct textblock on a page that may very well be composed of multiple objects. Asthe reconstituted structure analyser is effectively the same object thatwas used for annotation create, it ‘understands’ the target object type.The core module calls the structure analyser which finds all possiblecandidate page objects matching the target object type and returns thisset to the core module. This is done rather than attempting to find theexact page object, for example by position within the DOM tree, as thetarget page object in the current instance of the web page may havemoved due to the addition or deletion of other page objects since theweb page was annotated. If the structure analyser fails to find anycandidate objects, the returned set will be empty and the core modulestops the annotation restore process at this point. In this case theprogram will not be able to attach the annotation and must degradegracefully. If the structure analyzer finds just one object of thecorrect type then this may be the object to which the annotation wasattached but the structure analyzer does not perform any contentanalysis and so this one candidate target must be passed on to thecontent analyzer for confirmation. If there are multiple matches by typethen these objects are all valid candidates and the structure analyzerreturns the set of objects which must be filtered to find the correctcandidate. By this means of selecting page objects by type the programdetermines any viable matches for the annotated object.

The second step in the annotation restore process, performed by thereconstituted content analyzer, is to filter the set of candidate targetobjects to identity the actual target. The core module calls the contentanalyser passing in the set of candidates found by the reconstitutedstructure analyzer. The content analyser scans each candidate in turn tofind a match using the restored target locating data. The precisealgorithm for implementation of this matching process will depend on theobject type. If there is no match, the content analyzer returns a ‘matchfail’ result to the core module which stops the annotation restoreprocess at this point.

The third step in the annotation restore process is performed by thereconstituted helper which recreates the annotation content andannotation point using data from its restored state. The core modulethen manifests the annotation content at a location (spatial and/ortemporal) determined using the target point, annotation point andrestored association data.

It will be recognized by those skilled in the art that the annotationrestore process described above may be used to match target locatingdata with data extracted from candidate objects of whatever type andthat by this means targets may be identified whatever the media type andwhether the target be the page, a page object, a collection of pageobjects, a region of a page object or a point on a page object. It willfurther be recognized that the process may result in cases in whicheither the target is not found or that multiple matches are found. Ifthe page has changed so that the target is no longer on the page or thatcharacteristics of the target have changed radically then it may be thatthe target cannot be found. It is also possible that the page may changeso that there are multiple precise matches and the semantic analysercannot determine which is the correct match. Multiple recovery policiesare possible at this point in order that the program degradesgracefully. For example, the multiple options may be presented to theuser or to the creator of the annotation.

Should the page content change during manifestation, annotations whichwere previously manifested may no longer be valid due to changes to, orremoval of, the target object. Also, annotations which previously couldnot be manifested due to failure of the reconstituted structure orcontent analyzers to find a match may now be manifested due toappearance of the target object or content. In the event that the webpage changes, the core module will re-invoke the restoration analysisprocess to test if the web page change resulted in the appearance (ordisappearance) of a page object matching the target locating data type.The DOM provides several events for the purpose of notifying about DOMstructure changes.

The form and content of the target locating data is specific to eachtype of analyzer and comprises sufficient data to allow the analyzer tofind the original target. The amount of data that a content analyzerwill need to store in order to be able to positively locate the originaltarget will vary and the purpose of the decoupled architecture describedin this embodiment is to allow refinement of the content analysisprocess without invalidating annotation tools already implemented andannotation sets already saved. This should not however be taken to implythat monolithic versions of the program could not be implemented thatwould be able to operate without invalidating existing tools and savedannotation sets. The detailed treatment of algorithms for analysingcontent and content matching is also outside the scope of thisembodiment description, but several examples will be given to illustratethe process.

A simple example of the annotation restore process is to find a singleword or phrase within the correct text block on a web page that may verywell be composed of multiple text blocks. The target locating data willconsist of the target word or phrase plus additional data from theannotated text block to provide context for the target. This contextcould be a number of preceding words and following words. The structureanalyser simply locates all text objects on the manifested web page.When the content analyser scans each of the candidate target objects, itfirst eliminates those that do not contain the target word or phrase.Then it uses the context data (preceding and following words) toeliminate text blocks that do not contain the correct context. Anyremaining candidate target objects are matches for the target locatingdata. If there is only one match, then this is the target for theannotation and the target point may be calculated. If there are multiplematches the restore process has failed and the program degradesgracefully.

In a second example, the target may be a location within an image. Inthis case, the target locating data will consist of data about the imagein which the target is to be found. Such data could include a calculateddigest of the whole image data plus the location of a single pointrelative to the origin (top left corner) of the image. ‘Digest value’ isa term familiar to those skilled in the art.

In a third example, the target may be a region within an image. In thiscase, the target locating data will consist of data about the image andabout the image region in which the target is to be found. This type ofimage content analyzer would be used, for example, if the image contentis expected to change slightly yet the region the annotation is attachedto will remain the same. The region specific target locating data allowsthe target to be found through use of fuzzy image matching. Examples ofsuch data could include the dimensions of a region (width and height).calculated digest of the region, calculated digest(s) of sampledsubset(s) of the image (e.g. 10 pixel block samples at known positions).

Those skilled in the art will recognize that such target locating datamay be collected and saved for the restore process for any type and thatthere are many variables for any type which may be so saved to lateridentify the correct candidate. They will further recognize that whilstit is impossible to create an algorithm which is one hundred percentfoolproof it is possible to devise algorithms which are highly reliable.Furthermore, such algorithms may use such techniques as fuzzy logic todeal with borderline cases and improve performance. The architecture ofthe described embodiment enables the application of appropriate fuzzymatching logic to web pages and objects sourced from different websitesand also enables fuzzy logic content analyzers to be deployed with noimpact on annotation tools.

If the content analyzer cannot find a match the target cannot berestored. There are several possible reasons for a failure to match:

-   -   the original content no longer exists on the displayed web page        or the content has changed materially    -   the content is within an element not yet updated by JavaScript    -   the content is not yet loaded

In the first case, the annotation cannot be displayed and the user maybe notified via an information dialog. This case is discussed furtherbelow and necessitates graceful degradation. In the latter two cases,the annotation may be displayed only when the content has been updated.As described above, the DOM supports several events for notificationwhen content or attributes are changed. The core module tracks theseevents on page objects of interest to detect when the page objects havebeen modified—for example, when a JavaScript managed scrolling regionhas new content or when a new URL is loaded into an <img> element. Thecore module uses these events to determine if:

-   -   a displayed annotation should be removed if the content to which        it was attached has now been removed    -   an annotation which previously could not be displayed because of        a content match fail may now be displayed    -   a currently displayed annotation must be moved because of        content change

When such events are received, the core module invokes the semanticanalyzer module again to search for the target.

By the above described means the invention will use the target locatingdata to accurately find targets and will be able to reconstruct theoverwhelming majority of annotations in appropriate semanticrelationship with the underlying page if the only change is to pagelayout. If the target is still there, the structure and contentanalyzers will find it and the helper will render the reconstitutedannotation.

By this means too, the invention will also be able to accuratelyreconstruct a high percentage of cases of annotation even if theunderlying content has been changed. Because algorithms may be devisedto recognize page objects and targets within predetermined bounds ofstatistical accuracy as the page changes the program is tolerant ofchange across a great deal of the page. It will be evident to thoseskilled in the art that by this means the algorithms used to select datapoints for page object selection and recognition and target selectionand recognition may be made extremely sophisticated and it will also beevident that the algorithm(s) selected are not material to thefeasibility of the invention.

If the user has permission to edit any of the restored annotations,visual feedback is used to indicate which of the annotations may bemodified. Should the user choose to modify an annotation, the editor isinvoked to handle changes to the content and the core module supportschanges to the annotation data in the same way as when the annotationwas created. A modified annotation set may be re-saved to the serverallowing all other users with permission to view the annotation set toaccess the updated annotations.

The described architecture enables the invention to annotateappropriately in ways never before even considered. For example, say aweb page has a carousel of five images that cycles through the imagesand our user selects one of the images and annotates it then the programwill only display the annotation when the appropriate image isdisplayed. The content analyzer recognizes that the other images are nota match (e.g. by comparing their digest value with the digest valuestored when the target was defined) and signals the helper via the coremodule to only display the annotation when the correct page object(image) is being displayed.

If the page has changed so radically that the program cannot confidentlyfind the target it must degrade gracefully. For example, if the textcontent analyzer cannot find the specific set of words to which theannotation was originally saved, it uses its stored signature data toattempt to find the closest phrase and, if that fails, it returns thelocation of the content object itself. If nothing can be found, thecontent analyzer returns a status to indicate that the annotation mediacannot be placed. This may be because the web page has changeddramatically and the original content object has been removed entirelyor changed to such an extent that it cannot be recognised or the objectis too large for its allocated display area and so must be scrolled. Ifthe page has changed so radically that no match can be found the programinforms the user that one or more of the annotations cannot berecreated. This result is then saved on the server and can becommunicated to the creator of the annotation who may elect to reworkthe annotation. By this means the program degrades gracefully.

It will be appreciated by those skilled in the art that the abovedescribed process represents an annotation manifestation means formanifesting restored annotations with semantic integrity on a secondinstance of the web page.

It will further be appreciated that whilst the above described processtalks of ‘the user’ it is essentially the same for a group of userssharing the process of creating, editing, saving and restoring anannotation.

(f) Annotation Sharing

Once the annotation data is stored in a database on the server this datamay be used to provide further functionality to users. The annotationsmay be queried in accordance with multiple criteria; for example, theuser who created the annotation, the date of creation or the websiteannotated. Because the database may be searched by any registered userand annotations may be retrieved by that user (subject to accesscontrol) annotations may be shared by members of the community. Thisability to support the sharing of annotations amongst community membersmeans that annotations are social objects.

Those skilled in the art will recognize that there are many communitywebsites organized around social objects. Examples of such websitesinclude flickr (photos), youtube (videos) and facebook (statusinformation). Such sites provide sophisticated facilities to browse andsearch such objects, communicate with other members of the community andview interesting site statistics and recommendations. The inventionenvisages such a community organized around annotations. Those skilledin the art will recognize that the technology for implementing suchcommunity functionality is relatively mature and there are noinsuperable technical obstacles to developing such community supportfacilities.

The sharing of annotations may take place on a website via a websiteadd-on or URL redirect, as is familiar to those skilled in the art. Itmay also take place by means of the functionality of the browserextension as described above.

(g) Developing New Annotation Tools and Annotation Tool Support

The process of creating a new tool to create and edit annotationsdepends upon whether the program already supports the types orstructures required by the new tool. In the event that the types andstructures are already supported the process is relativelystraightforward. The developer creates the tool icon, media (ifappropriate), editor and helper. This tool can access support from theprogram via the Core API and SA API. This support means that for theexisting types and structures already supported by the program andrequired by the new tool, accessing this support and thereby accessingthe ability to work with events and semantic integrity is alreadysupported and hence this implementation burden is removed from thedeveloper. It is anticipated that as a relatively small number of mediatypes are used on a very large number of web pages many tools may bewritten which will have all required type and structure support.

In the event that unsupported types are required by the tool it isnecessary to put such support in place by implementing additionalstructure and/or content analysers. This process is one of developing asmany new analyzers as necessary. It will be recognized that havingcreated such support for new types this support is now available forfuture tools, thereby further extending the support platform.

Once the new tool has been created and any required tool support is inplace all that is required is to put in place the necessary toolconfiguration information and register the tool with the server. Suchconfiguration information includes: name, icon, default target, defaulttiming, default event handling, type selector, page source URL pattern,page object URL pattern, any information related to the sale of toolse.g. price. The server allocates a tool identifier and adds this to theconfiguration information when the tool is registered.

Those skilled in the art will appreciate that the above described meansof creating and installing new tools is a well defined process with thefollowing advantages: (a) it is easier for developers to create toolswith semantic integrity (spatial and temporal) and event handlingsupport, (b) new types and structures may be readily accommodated andaccommodated in a way that extends the program to further ease futuretool development.

(h) Annotation Programming

As discussed above, the behavior of an individual annotation may beprogrammed by means of the Annotation API. And as also noted above, theMCD includes the ability to respond to and emit events and control whichannotation attributes are exposed for modification. Hence annotationscan respond to and emit events to influence the behaviour of otherannotations. Such dynamic behaviour may be implemented in the annotationhelper and/or in the annotation manifestation data (for example as aFlash ActionScript). By this means an annotation may, for example, bedisplayed on the screen and then await user input, such as clicking onan area of the annotation display space. On receiving notification ofthis event the annotation may then trigger other action(s) by, forexample, manifesting a different annotation content. If annotationbehaviour is implemented in the manifestation data, the helper providesan interface to the annotation event mechanism.

Control scripts are specialised annotations that have no visualmanifestation and consist only of programmable content which comprisesblocks of actions associated with an event. Actions are executed insequence and comprise one or more of (a) sending an event, (b) modifyingannotation triple data. The triggering event data payload, comprisingkey/value pairs, is used to construct a new event or to update anannotation triple. It will be recognised by those skilled in the artthat such action scripts can be expressed as either a text based or avisual programming language and choice of expression is not material tothe implementation of the event/action mechanism. Alternativeimplementations of the control tool may be provided with differentcontrol script expressions.

FIG. 12 shows a simple example of how annotation programming andcommunication between cooperating annotations can be used to implement adynamic manifestation coordinated by communication via the core module1802. The annotation set comprises three annotations: 1804 whose contentis a bitmap image which is attached to the center of an image on the webpage, 1806 a flash movie which is attached to the whole web page and1808 a control annotation. Execution of code within the core module isrepresented by 1803, execution of code within the helper for annotation1804 is represented by 1805, execution of code within the flashActionScript and helper for annotation 1806 is represented by 1807 andexecution of code within helper for annotation 1808 is represented by1809. Although FIG. 18 depicts execution control as being transferreddirectly between 1803, 1805 and 1809, the mechanism for thiscommunication uses events which are dispatched via the Annotation APIand routed via the core module. Annotation 1804 is initially not visible(its MCD specifies that the annotation's visibility attribute should beset to ‘hidden’), the flash animation of annotation 1806 is configuredto not auto-start and control annotation 1808 starts as soon as theannotation set is manifested. The animation of 1806 comprises a numberof alternative elements all of which depict a stick character carrying ahammer; one element has the character walking from left to right andanother element has the character walking from right to left. At the endof each sequence, the stick character stops walking and strikes thehammer onto a point near its feet. The flow of event informationco-ordinating this simple sequence is as follows. The core module 1802instantiates the annotation set with member annotations 1804, 1806 and1808 and sends the start events 1810, 1812 and 1814 to these memberannotations respectively. The control annotation 1808 has a controlscript that immediately uses the Core API to discover the web pagewindow dimensions and the location of the currently non-visible bitmapimage annotation 1804. The control annotation 1808 then uses thisinformation to determine whether the currently non-visible bitmap imageannotation 1804 is nearer to the left or right hand edge of the window.The control annotation 1808 then dispatches an event 1816 with payloadcomprising the window dimensions, the keyword ‘left’ or ‘right’ toidentify the edge of the browser window furthest away from the currentlynon-visible bitmap image annotation 1804 and the X,Y coordinates of thisannotation 1804. The currently halted flash animation annotation 1806 islistening for the event and, on receipt of the event, uses the payloaddata to resize its display ‘canvas’ and select the appropriate animationsequence to show the stick character walking from the edge of the screentoward the location of the currently invisible bitmap image annotation1804. When the stick character reaches the location of annotation 1804,the flash animation annotation 1806 runs its hammer strike animation anddispatches another event 1818 to notify the control animation 1808 ofthe animation sequence termination. On receipt of the event, controlanimation 1808 dispatches two events: the first 1822 is sent to thecurrently non-visible bitmap image annotation 1804 and causes it to bemade visible and the second 1820 is sent to the terminated flashanimation animation 1806 and causes it to be made invisible. By thismeans, in this example, the core module 1802 invokes three cooperatingannotations 1804, 1806 and 1808 to create the described effect.

Although events are shown as travelling directly between annotations,their physical path is always via the core module which is responsiblefor event distribution. The annotation MDC specifies which named eventsan annotation may generate and which events the annotation wishes toreceive. The Annotation API defines an event handler registration methodwhereby all events (both core generated and originated by otherannotations) may be received and processed within a single helperfunction.

By the above described means, the set of annotations on a particularpage may co-operate to create a dynamic manifestation controlled byevents. These annotation events may also be propagated via the networkto annotations manifested on other user's computers. The annotationserver provides services to authenticate and route such inter-annotationcommunication.

Annotation set data may also be accessed and modified, for example to:create or delete annotations; or to search for an annotation using aquery on the annotation data (e.g. get the set of annotations createdusing a particular tool). Hence it will be appreciated by those skilledin the art that annotation sets are also programmable.

It will be understood by those skilled in the art that the provision ofthe API means that the “user” of the program may be human or a computer.A human user will interface with the program via the core module thatenables interaction to undertake such tasks as to select an annotationtool, use it to create or modify an annotation, to fix the target, tosave the annotation, to restore it and to edit it and resave it. Acomputer may also interface with the program, but will not require thesame interaction modules required for a human user. It will undertakeannotation tasks by accessing the API. For example, a human user caninteract with the program by using a mouse to move a cursor to move anannotation around the screen and move crosshairs to identify theannotation target whereas a computer can access the API to select anannotation, edit its content and locate it and define its target. It canfor example, define the target by expressing it in a Target LocatingData Specification Language. Statements in this language are interpretedby the semantic analyzer module to select an appropriate set ofstructure and content analyzers to explore the web page for a matchingpage object and, if required, specific page object content.

(i) Functional Independent of Program from Annotated Websites/Pages

As noted above, the document system and the annotation system aredistinct systems. The document system can exist without the annotationsystem and would be entirely unchanged if there were no annotationsystem in place, which is to say that no components of the system wouldbe changed and no aspects of the functionality of the system would bechanged. Similarly the annotation system demands no changes of thedocument system for its operation; no components of the document systemneed by changed and no aspects of the functionality of the documentsystem or of the documents it manages need be changed. The two systemsneither share API's nor state by contract and as will be discussed belowtheir only point of common contact is the DOM API.

As described earlier, FIG. 2 depicts a document system 100 and anannotation system 200 manifesting via the rendering engine of a commonDOM implementation within a web browser 120. Those skilled in the artwill recognise that these two systems are distinct and whilst they maybe implemented in various ways with more or less complex combinations ofservers and communications technology such differences do not change thebasic functionality or distinct nature of these two systems.

This is an important point which warrants further elaboration: asdescribed herein the means by which the web page is annotated is thatthe program accesses the DOM to analyse the content of therepresentation of the page to be annotated, makes use of the results ofthis analysis to interface with the user as he creates and editsannotations, and accesses the DOM to integrate the requiredrepresentations of the user generated annotations with the web pagerepresentation resident in the DOM. The rendering engine then manifeststhe composite representation of the annotated page. The two systemswhich result in an annotated document (web page) share some resources asnoted above (the client computer, the web browser, the communicationchannel) but their use of these resources is distinct and separate. Theyoverlap only in their access to those objects accessible by the DOM.Those skilled in the art will therefore recognise that these systems areindeed functionally distinct and independent. This property of theprogram is hereinafter known as ‘functional independence.’ Those skilledin the art will appreciate that there can be no functional independenceas defined herein if the document system and the annotation system sharestate of any kind by contract. In the described implementation the twosystems share no such state and are truly functionally independent.

The program's functional independence extends to embedded objects in XMLdocuments. Such objects are referenced in the DOM. The programrecognises such references, analyses the object in question and suchcontent as is served by such an object, and integrates any appropriateannotation XML fragments with the content in precisely the same way.

It will further be recognized by those skilled in the art that the abovedescribed implementation requires no modification of the documentsystem. Specifically, it requires no changes to the website whosecontent is being annotated. It requires neither changes to the softwaresupporting the website nor to the content of the website, nor does itrequire any special permissions from the site publisher. Finally, itwill be noted that the system does not copy or store a copy of thecontent from the document system.

It will be recognized that, whilst the described implementation achievesannotation with functional independence there are alternative means ofannotating documents with functional independence. It would, forexample, be possible to edit documents in the document buffer of thebrowser or even to intercept and manipulate the representation of therendered page. It will further be recognised that the described processof semantic analysis which results in the annotation triple with thelocating data necessary to provide semantic integrity is consistent withsuch a different means of providing functional independence.

(j) Annotating Documents Other than Web Pages

The embodiment described above is for web page annotation. Those skilledin the art will however recognise that the components, mechanisms andworkflows described above can equally well be applied to annotation in aproprietary application (one for which the interfaces and protocols tointernal operations are not published and freely available to thirdparty developers rendering the application a closed rather than openenvironment) that uses XML for document structure and in which thedescribed components, mechanisms and workflows must be expressed in theprogramming language of the proprietary application and compiled intothis closed application. An example of such a proprietary application isa word processor that uses XML to encode its document files. A good useof the annotation described above in this example would be to supportmulti-author document creation: annotations can be used to placecomments on existing document content to make suggestions for changes orto mark revisions. A further example is one in which the applicationuser interface is described using XML allowing the user interfacepresentation and/or semantics to be altered—a procedure known to thoseskilled in the art as ‘skinning’ The components, mechanisms andworkflows described above can be applied to such an application allowingmodification for accessibility or for integration into a larger system.

(k) URL Mapping

It will be recognized by those skilled in the art that a web page for aparticular URL may display different content for independent orsuccessive manifestations of the web page. Such content may bedetermined according the ID of a logged-in user or may be determined bythe execution of in-page JavaScript. The latter mechanism is oftenemployed for performance reasons such that some websites redraw subsetsof a page rather than the whole page to present what the user sees as anew page. An example of such a website is www.facebook.com that makesuse of what it terms ‘BigPipe’ to redraw subsections of pages that itterms ‘pagelets.’ (BigPipe is described in a Facebook Engineering Notes:http://www.facebook.com/note.php?note_id=307069903919 andhttp://www.facebook.com/note.php?note_id=389414033919). This redrawnpage may have a different URL, and hence appear to be a new page, andyet it is in fact the same page.

The net effect of these mechanisms is that the structure and/or contentof the manifested web page is not directly related to the URL of thepage. In the above description it has been explained how the system usesthe page URL to recognise that a page has changed so that the system mayremove annotations, inform users about annotations on the current pageand add annotation as appropriate. In the case of sites where the URLmay change and yet the underlying page does not, however, the URL is nota sufficient indicator that the content has changed and hence therelationship of the annotation to the underlying content must bereviewed.

In the case of such websites the system generates a unique pageidentifier which encapsulates the web page state. This identifier isderived from an inspection of page structure and/or content and thealgorithm for its generation is dependent on the web site. Those skilledin the art will recognise that such identifier generation may beimplemented using an additional category of component, the URL mapper,instances of which are selected by pattern matching on the web site URLand invoked whenever the system requires a web page identifier—i.e.whenever the system detects that page content has changed. Support fornew web sites, for which there is a non-unique relationship between webpage URL and web page state, is achieved by creating a URL mapper forthe web site—in much the same manner as a web site specific structureanalyser may be provided.

(l) Applications of Annotation Widgets

The invention enables a new class of web widgets. Such widgets areannotations of the kind described above that may be apparentlyincorporated into any web page(s), and located on any desired locationon the page, offering the functionality of the widget to the user on thepage in question.

In the example below we consider the case of the web widget known tothose skilled in the art as the social plugin. Such social plugins areprovided by the social networking site provider e.g. facebook, areinstalled into publishers' sites and communicate with the socialnetworking site. By this means, a user of the social networking site mayfor example login to the social network from the publisher's site andinteract with social data exchanged between the publisher's site and theuser's computer but which originates in and is hosted by the socialnetworking site.

FIG. 13 is a high level diagram of the system architecture implied bysuch social plugins. A description of the operation of this system 1900follows. A user on a client computer 1902 interacts with a web browser1904 installed on the client computer 1902. This web browser 1904communicates with a web server 1910 on a website publisher servercomputer 1908. This communication takes place by means of requests 1906and responses 1914. This web server has two components: component 1912is the website software which generates the publisher's website andcomponent 1920 is the social plugin. For many interactions with thewebsite, the requests 1906 and responses 1914 may be satisfied entirelyby the publisher's website software 1912. For requests which requireinteraction with social data stored on the social network computer 1924further communication is required. If the website software 1912 detectsthat such social information is required that must be served by thesocial network computer 1924 then it communicates with the social plugin1920 installed with the website software 1912 on the website publisherserver computer 1908. It communicates with the social plugin 1920 bymeans of requests 1918 and responses 1916. The social plugin 1920, inturn, communicates with the social network computer 1924 by means ofrequests 1921 and responses 1922. When the social network computer 1924receives a request (for social network data or an update to socialnetwork data) this request is communicated to the social networkingserver 1928 and that request is given access to that segment of thesocial network data 1926 which is available to social plugins. Not allsocial network data is accessible to social plugins in this way. 1930represents social network data which is accessible directly by usersinteracting with the social networking site but which is not accessibleby means of the social plugin 1920. Those skilled in the art willrecognise that the proportion of such data that may be made availablemay vary. When the social network server 1928 has satisfied the requestit communicates its response 1922 to the social plugin 1920 which inturn communicates the response to the website software 1912 whichcommunicates its response 1914 to the user's web browser 1904 on theuser computer 1902. By this means, the system 1900 is able to makesocial data 1926 stored on a social networking website available on athird party publisher's server 1908 and hence to a user interacting withthe publisher's site. By means of such user interactions users may viewand update social data whilst on third party website (sites other thanthat of the social network).

Contrast the above with the example system architecture 2000 illustratedin FIG. 14 which employs a social plugin implemented in the form of anannotation as described herein. A description of the operation of thissystem 2000 follows. A user on a client computer 1902 interacts with aweb browser 1904 installed on the client computer 1902. In contrast tosystem 1900, in system 2000 the web browser has an additional component:the annotation system's browser extension 2002. The user's interactionwith the web browser 1904 generates communication with a web server 1910on a publisher's website server computer 1908. This communication takesplace by means of requests 1906 and responses 1914. The website servercomputer 1908 hosts the website software 1912 which generates thepublisher's website, but in contrast to system 1900 it does not includethe social plugin 1920. Hence, in contrast to system 1900, the websitecomputer 1908 is unable to directly communicate with the social networkcomputer 1924 and thereby unable to directly access social data 1926.The publisher of website 1912 need have no knowledge of the existence ofsocial network server 1928. For those interactions between the clientcomputer 1902 and the website server computer 1908 that do not requiresuch social data the requests 1906 and responses 1914 may be satisfiedentirely by the publisher's website software 1912. Such communicationscannot however access the social data 1926 because this data is nothosted on the website server computer 1908 and, as noted above, thewebsite server computer 1908 cannot communicate directly with the socialnetwork computer 1924 to access this social data 1926. For userinteractions which require interaction with social data stored on thesocial network computer 1924, as will be described below, suchcommunication is routed through the annotation server 2004. Userinteraction with social data is managed via an annotation which iseither created and placed by the user or is part of an annotation setthat the user has selected for manifestation. Requests for socialinteraction are generated by the annotation and routed via the browserextension 2002 which detects that information is required that must beserved by the social network computer 1924. The browser extension 2002then communicates with the annotation server 2006 hosted on theannotation server computer 2004. Such communication takes the form ofrequests 2050 for social data and for notifications of updates to socialdata and of responses 2052 to such requests. Any such request 2050 isrouted by the annotation server 2006 to the annotation server socialmedia extension 2010, which communicates the request 2012 to the socialnetwork server 1928 on the social network site computer 1924. The socialnetwork server 1928 accesses the social data resource 1926 for update orretrieval. The social network server 1928 then sends a response 2014 tothe annotation server social media extension 2010 on the annotationserver 2006 and this response is forwarded as a response 2052 to thebrowser extension 2002 on the user computer 1902. The browser extensionthen uses the Annotation API to forward the result of social mediainteraction to the requesting annotation for manifestation on the user'scomputer 1902. By means of such user interactions, users may, as withsystem 1900, view and update social data whilst on third party website(sites other than that of the social network). As the figures make clearthere is a striking difference between system 1900 in FIG. 13 and system2000 in FIG. 14. In the latter case the plugin does not have to beinstalled on the publisher's website. Instead of every publisher thatwishes to support such social interaction having to install a socialplugin on every page on every website by means of social plugins, allthat is required is the browser extension be installed on usercomputers, a social media extension installed on the annotation serverand provision of social annotations. The annotation server thereaftercan place the social content from the social media site ‘on’ any webpage on any publisher's site. The term of art used to describe suchactive content ‘placed’ onto a publisher's website is a ‘widget’. In theexample of FIG. 13, such widgets are integrated into the websitecontent. In the example of FIG. 14 such widgets are annotations. Theuser on his client computer who elects to see such widgets and who viewsa page on the publisher's site will then see the annotation widget andmay interact with this novel form of social plugin. His interaction isrouted though the annotation server to the social network site. As thoseskilled in the art will recognize, there are alternative architecturesemploying such annotation widgets, but all may obviate the need fordirect integration of the social plugin into the publisher's website.

The use of this novel annotation widget requires no editing of the webpage code and does not even require the permission of the publisher. Itwill be recognized by those skilled in the art that social plugins areone example of a whole class of widgets or other rich media objects thatmay be apparently incorporated onto web pages.

A further such example of a novel application of annotation widgets isin advertising. As noted above, annotation widgets as described hereinmay be placed anywhere on any web page without requiring any cooperationfrom the page publisher. Hence it is possible to place advertisements onany web page. Furthermore, because such advertisements may employ any ofthe capabilities of the annotations described herein they may have novelproperties. For example, advertisements may include branded contentwhich may be copied from the ad and used as content for annotations,thereby virally spreading advertising content.

In a further such example, annotation widgets may be used to implementgames which could be available at all times on all web pages. The gamecontext and play behavior may include content from the annotated webpage accessed via the SA API. Through use of USI and inter-annotationcommunication via the Annotation API, the game play may encompass manyweb pages. Because annotations may be shared such games could bemulti-user. And because they can incorporate social content, asdescribed above, they can be social games.

(m) Divisions of Labor

As noted above, as is obvious to one skilled in the art, the program maybe implemented with different divisions of labor between the browser,the browser extension and the server.

It is helpful to separate the annotation process into creation andconsumption phases: creation is the set of tasks whereby annotationtools and semantic analysers are used to create the annotation;consumption is the process of the user experiencing the manifestation ofthe annotated document. Creation may involve user interaction or it maybe performed solely under the direction of an application using theAnnotation API. In the former case, a browser client component isrequired to support interaction with the user. In the latter case, aclient component is not mandatory as all of the functionality to supportthe Annotation API may be implemented on a server. A client component ishowever essential for the consumption phase,

We may distinguish between those instances wherein a human userinteracts, with the annotation(s) and those cases where the user doesnot interact and simply passively experiences the manifestation of apreviously created annotation.

i. User Interacts with Annotation.

In this case there must be a program component on the client to supportsuch interactions. Hence the functionality of the client side of theprogram must support programmable annotation content, for examplewritten in the JavaScript language. Such programmable content willreceive user input and perform one or more of: modify annotation contentmedia; interact with other annotations via the Annotation API. As thoseskilled in the art will recognise, a possible implementation topologywould be to support all annotation rendering on the server, includingexecution of such programmable annotation content. In this case aminimal client layer would nonetheless be required to forward all userinput to the server.

ii. User Passively Experiences Annotation Manifestation.

In this case the client is simply being used as an output device tomanifest annotations composed on the server. Hence the functionality ofthe client side of the program need only support rendering of theannotation media.

If we now consider how possible divisions of labor between the clientand server sides of the program we can discuss two boundary cases:

i. Maximal Implementation in Client

In this case the program functionality in the client supports userinteraction for the creation and modification of annotations andannotation sets. It further supports, in a manner which is functionallyindependent of the document system components such as the documentserver, all functionality required for semantic analysis and semanticintegrity and the interaction with such annotations not just by means ofthe client program but by other programs by means of the Annotation API.

The server functionality in this case enables storing, searching andbrowsing such annotations and thereby the presentation of annotations asa social object to a community who share annotations.

ii. Maximal Implementation in Server

If the user is only passively consuming annotations then the program inthe client supports only the functionality to manifest annotationscomposed on the server. If the user needs to interact with annotationsthen the program in the client must further provide a proxy for inputand functionality executing on the server.

The server would in this case include all functionality required forsemantic analysis and semantic integrity and the interaction with suchannotations not just by means of the (minimal) client program but byother programs by means of the Annotation API. It would support theability to create a composite annotated document from the document beingannotated and the annotation data and the ability to transmit thiscomposite to the client for display. It would further support thestoring, searching and browsing of such annotations and thereby thepresentation of annotations as a social object to a community who canshare annotations.

It will be understood by those skilled in the art that the abovedescribed boundary cases represent extremes and that there are manypossible divisions of labor, between the client and server sides, of theabove described program functionality. It will further be recognised bythose skilled in the art that any such redistribution of labor is simplyanother implementation of the program. At any given time thedistribution of functionality of the implementation may vary, dependingupon technical factors such as the software and hardware available onclient computers, the bandwidth of prevalent computer networks, and thepower and economics of server computers.

(n) Alternative Document Encoding Schemes

Whilst XML is the dominant mechanism for encoding documents and documentfragments, at the time of writing there are alternatives, for exampleJSON. There are currently document systems wherein the document serverresponds to requests by sending JSON encoded data to the client wherethey are processed by JavaScript to transform the JSON into a form whichmay be entered into the DOM by means of the DOM API.

Those skilled in the art will however recognise that whilst much of theabove description refers to XML formatted documents the describedimplementation is in fact agnostic with respect to encoding schemes. Anydocument, whatever encoding scheme is employed, must at some point berepresented in the DOM and it is only at this point that the programfunctionally intervenes to annotate the document. The program functionsnot by operating on the encoded document but by manipulating DOM datastructures (for example by inserting DOM fragments representingannotations) to create a composite of the original document and theannotation which is rendered and displayed. Hence whilst theimplementation described above has talked of XML documents the format ofthe document prior to the DOM is irrelevant to the program.

(o) Browser Tabs and the Browser as Target

Whilst the above implementation has described how the page and parts ofor points on the page may be the target of annotations there are twoother targets:

1. a browser tab session

2. the browser session

An annotation attached to either of the above targets will be persistentacross all manifested web pages and will terminate only when:

1. deleted by user interaction by interaction with the control menu

2. deleted programmatically by means of the Annotation API

3. the user closes the browser tab to which the annotation is attached

4. the user terminates the browser session to which the annotation isattached

Those skilled in the art will recognise that the above targets mayreadily be implemented by extension of the mechanisms described above.

(p) Alternative Implementation of Annotation Linking

It will be recalled that in the above described implementation theserver allocates a unique identifier to the stored annotation set: theUnique Set Identifier (hereinafter ‘USI’). This unique resourceidentifier enables:

-   -   the annotation set creator to later access, download and update        the set or to delete it if required    -   annotation set sharing    -   annotation set linking

It will be understood by those skilled in the art that this aboveimplementation implements linking at the level of the annotation set. Inthis implementation it is annotation sets, rather than individualannotations, that are linked to the underlying content. There arehowever other implementations which link individual annotations ratherthan annotation sets to the underlying content. Directly linkingannotations rather than annotation sets can also implement a linkedannotation space ‘on top of’ the world wide web. Hence, as in the abovedescribed implementation, users may navigate from annotated web page toannotated web page by clicking on displayed annotation link objects.This implementation is perfectly viable and is described below.

In this implementation the server allocates a unique identifier to eachstored annotation: the Unique Annotation Identifier (hereinafter ‘UAI’).This unique resource identifier enables:

-   -   the annotation creator to later access, download and update the        annotation or to delete it if required    -   annotation sharing    -   annotation linking

In this case the annotation link object is a web page <anchor> elementwith a UAI as its ‘href’ attribute value). If, for example, a user is onthe page www.zzz999.com/456 and actions a UAI link on this page then theuser's browser will change the displayed page from www.zzz999.com/456and will instead display the web page and associated annotationreferenced by the UAI. This linked-to new page could be any properlyaddressed web page e.g. www.zzz999X.com/678.

FIG. 18 is a high level representation of the relationships between aUAI anchor 2410, the annotation data 2430 of the individual annotationit refers to and the document 2450 annotated by that annotationdescribed by annotation data 2430. As the UAI is a resource identifierstring, just like a URL or URI, it may be used as the target of ahyperlink (HTML element <a>). Thus, UAI anchor 2410 is an HTML <a>element embedded within web page object 2416 on web page 2418. The UAIhyperlink destination is a reference 2420 to annotation data 2430 in anannotation store 2432. The annotation data 2430 includes a reference2440 to the annotated document 2450 held in document store 2452.

Traversal of an ‘annotation link’ is actioned by an annotation helpereither as the result of user interaction with the annotation content oron receipt of an annotation event. The core module uses the UAI torefresh the browser display with the referenced web page and immediatelydisplays the annotations from the annotation set.

Annotation links use the same universal resource name format as webpages but with a unique scheme name to identify the protocol forretrieving web pages and associated annotations. The ‘Annotation AccessProtocol’ or ‘AAP’: for example the UAI string would be of the form:aap://annotation_id. The core module installs a handler for the ‘aap’scheme so that any reference to a UAI is handled by the program. It willbe recognised by those skilled in the art that a different scheme namemay be used without compromising the architecture and that the schemename must be registered with LANA (the Internet Assigned NumbersAuthority).

Those skilled in the art will recognize that in the case of the abovedescribed alternative implementation, annotations may be linked todocument elements which may themselves be incorporated by reference.

Although it is possible to see a single annotation as a case of theannotation set wherein there is only one member this is not strictlyaccurate. The annotation set with only one member still incorporatesannotation set data which would not be necessary in the case of thedescribed alternative implementation for linking individual annotations.

The USI and the UAI are both means of referencing information aboutannotations. In the case of the USI, a set of annotations. In the caseof the UAI, an individual annotation. The USI and the UAI may beconsidered cases of the more general notion of a Unique Annotation LinkIdentifier (hereinafter ‘UALI’).

(q) Other Embodiments

A further alternative implementation exploits the linking of annotationsto page elements. Annotations which are associated with page elementscontaining content associated with a URL other than the page URL areimplicitly linked to that content via the Target Locating Data. Bymaking this link explicit and storing as an accessible part of theAnnotation Data, it is possible to query for individual annotationsusing the URL.

When a new web page is loaded into the browser, the core module uses astructure analyser to collect all URLs embedded within page elements.This list is used to query the annotation server for annotation sets andindividual annotations linked to the displayed web page and its content.The user is presented with the results of the query and they can chooseto manifest an annotation set and/or combinations of individualannotations.

The described program is capable of many further variations in detailedimplementation that can be derived from the description contained hereinby a person skilled in the art. All such variations and modificationsare considered to be within the scope and spirit of the presentinvention as defined by the following claims.

It is to be understood that although the invention has been describedabove in terms of particular embodiments, the foregoing embodiments areprovided as illustrative only, and do not limit or define the scope ofthe invention. Various other embodiments, including but not limited tothe following, are also within the scope of the claims. For example,elements and components described herein may be further divided intoadditional components or joined together to form fewer components forperforming the same functions.

Embodiments of the present invention have a variety of advantages. Forexample, embodiments of the present invention address the semantics ofannotation. More specifically, embodiments of the present inventionintroduce a well defined process of semantic analysis to analyse the webpage being annotated by a user, determine the location on the page thatthe user is attaching the annotation content to, and generate datasufficient to find this location in later instances of the page (despitechanges in page layout and content). This ability of semantic analysisto find the location that the user placed the annotation is the key toassociated property of semantic integrity which ensures that whereverpossible annotated web pages will appear with the user's annotations inthe correct location (in time and space) as intended by the annotatinguser so that the meaning of the user's annotation is preserved. Thisprovision of semantic integrity, independent of annotation type, dealingwith spatial and temporal integrity, is extensible to new media typesand styles of annotation. As changes to a web page become more profoundthe program degrades gracefully and in the event the original pagechanges out of recognition the program informs the user that it cannotmeaningfully render the previously annotated page. This extensibleprocess of semantic analysis to analyze non-text page content and theassociated provision of general semantic integrity is novel to theinvention.

Another advantage of embodiments of the present invention is that theyprovide support for new media types and tools. Whereas annotation toolsin existing systems have very limited functionality, embodiments of thepresent invention support tools that can access and annotate allelements and media types of any web page with semantic integrity. Theframework supports tools for example to add animation, edit audio,change images, spray paint over the page, change links and applets; copyelements from the underlying web page or elsewhere and reuse suchelements at will; and annotate any page element (or region of any suchpage element) include elements of the page that have no visual/spatialcorrelate such as music that plays when the page is displayed. As it isimpossible to anticipate all media types that may become web pagecontent, and as one may reasonably anticipate that annotationrequirements will continue to evolve, embodiments of the presentinvention provide an extensible media model and a Tools API so that theinventor or third parties may design, build and plug in new tools toaddress the requirements of new needs or media types as they emerge.

Another advantage of embodiments of the present invention is that theyenable annotations with radical new properties that redefine what ispossible of annotations.

Yet another advantage of embodiments of the present invention is thatthey enable temporal annotations. Such annotations include, for example,annotations with semantic integrity that appear and disappear at theright time(s) as intended by the user.

Yet another advantage of embodiments of the present invention is thatthey treat annotations as programmable objects all properties of whichmay be addressed and edited by means of an API. The provision of anannotation API eases the task of developers, who can build sophisticatedannotation applications with semantic integrity on top of the platform.Such developers do not need to know how semantic integrity is providedas the implementation is hidden behind the API interface.

Yet another advantage of embodiments of the present invention is thatthey enable annotations to be linked to and from along with theirassociated underlying web pages.

The techniques described above may be implemented, for example, inhardware, software, firmware, or any combination thereof. The techniquesdescribed above may be implemented in one or more computer programsexecuting on a programmable computer including a processor, a storagemedium readable by the processor (including, for example, volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device. Program code may be applied to inputentered using the input device to perform the functions described and togenerate output. The output may be provided to one or more outputdevices.

Each computer program within the scope of the claims below may beimplemented in any programming language, such as assembly language,machine language, a high-level procedural programming language, or anobject-oriented programming language. The programming language may, forexample, be a compiled or interpreted programming language.

Each such computer program may be implemented in a computer programproduct tangibly embodied in a machine-readable storage device forexecution by a computer processor. Method steps of the invention may beperformed by a computer processor executing a program tangibly embodiedon a computer-readable medium to perform functions of the invention byoperating on input and generating output. Suitable processors include,by way of example, both general and special purpose microprocessors.Generally, the processor receives instructions and data from a read-onlymemory and/or a random access memory. Storage devices suitable fortangibly embodying computer program instructions include, for example,all forms of non-volatile memory, such as semiconductor memory devices,including EPROM, EEPROM, and flash memory devices; magnetic disks suchas internal hard disks and removable disks; magneto-optical disks; andCD-ROMs. Any of the foregoing may be supplemented by, or incorporatedin, specially-designed ASICs (application-specific integrated circuits)or FPGAs (Field-Programmable Gate Arrays). A computer can generally alsoreceive programs and data from a storage medium such as an internal disk(not shown) or a removable disk. These elements will also be found in aconventional desktop or workstation computer as well as other computerssuitable for executing computer programs implementing the methodsdescribed herein, which may be used in conjunction with any digitalprint engine or marking engine, display monitor, or other raster outputdevice capable of producing color or gray scale pixels on paper, film,display screen, or other output medium.

1. A computer-implemented method for use with an annotation system:wherein the annotation system includes first data for annotating amanifestation of a first instance of a first XML document; wherein thefirst instance of the first XML document is served by a document system;wherein the annotation system is functionally independent of thedocument system; wherein the first data includes a first XML documentidentifier for the first XML document; wherein the first data isuniquely identified by a first Unique Annotation Link Identifier (UALI);wherein the document system is configured to respond to a requestcontaining the first XML document identifier with a manifestation of asecond instance of the first XML document; the method comprising: (A)receiving a first request containing the first UALI; (B) issuing asecond request to the document system, wherein the second requestcontains the first XML document identifier; and (C) in response to thefirst request containing the first UALI, manifesting the first data inconnection with the manifestation of a second instance of the XMLdocument.
 2. The method of claim 1: wherein the first data comprisesfirst annotation data representing a first association between: (a) afirst manifestation of first annotation content in connection with amanifestation of the first instance of the first XML document and (b) amanifestation of a first instance of a first target in the manifestationof the first instance of a first XML document; wherein the first UALIcomprises a first Unique Annotation Identifier (UAI) that uniquelyidentifies the first annotation data; wherein (A) comprises receiving arequest containing the first UAI; and wherein (C) comprises, in responseto the request containing the first UAI, manifesting the firstannotation data in connection with a manifestation of a second instanceof the first XML document.
 3. The method of claim 2, wherein the firstannotation data comprises first annotation content data representing thefirst annotation content, first target locating data for locating thesecond instance of the first target, and first association datarepresenting the first association.
 4. The method of claim 1, whereinthe first data comprises first annotation set data representing a firstannotation set; wherein the first UALI comprises a first Unique SetIdentifier (USI) that uniquely identifies the first annotation set;wherein (A) comprises receiving a request containing the first USI; andwherein (C) comprises, in response to the request containing the firstUSI, manifesting the first annotation set data in connection with amanifestation of a second instance of the first XML document.
 5. Themethod of claim 4, wherein the first annotation set further containssecond annotation data for annotating the first instance of the firstXML document; and wherein (C) further comprises, in response to therequest containing the first USI, manifesting the second annotation datain connection with the manifestation of the second instance of the firstXML document.
 6. The method of claim 1, wherein (A) comprises: (A)(1) ata client device: (A)(1)(a) receiving input containing the first UALI;(A)(1)(b) retrieving the first data from storage; (A)(1)(c) requestingthe second instance of the first XML document from the document server;and (A)(1)(d) manifesting the first data in connection with themanifestation of the second instance of the first XML document.
 7. Themethod of claim 1, wherein (A) comprises: (A)(1) at a client device:(A)(1)(a) receiving input containing the first UALI; (A)(1)(b)transmitting a request containing the first UALI to an annotationserver; (A)(2) at the annotation server: (A)(2)(a) requesting the secondinstance of the first XML document from the document server; (A)(2)(b)creating a composite of the first data and the second instance of thefirst XML document; (A)(2)(c) transmitting the composite to the clientdevice; and (A)(3) at the client device: (A)(3)(a) manifesting thecomposite.
 8. The method of claim 1, wherein (A) comprises: (A)(1) at aclient device: (A)(1)(a) receiving input containing the first UALI;(A)(1)(b) transmitting a request containing the first UALI to anannotation server; (A)(1)(c) requesting the second instance of the firstXML document from the document server; (A)(1)(d) transmitting the firstdata and the second instance of the first XML document to the annotationserver. (A)(2) at the annotation server: (A)(2)(a) creating a compositeof the first data and the first XML document; (A)(2)(b) transmitting thecomposite to the client device; and (A)(3) at the client device:(A)(3)(a) manifesting the composite.
 9. A computer system comprising atleast one computer-readable medium tangibly storing computer programinstructions, wherein the computer program instructions are executableby at least one computer processor to perform a computer-implementedmethod for use with an annotation system, wherein the method comprises:wherein the annotation system includes first data for annotating amanifestation of a first instance of a first XML document; wherein thefirst instance of the first XML document is served by a document system;wherein the annotation system is functionally independent of thedocument system; wherein the first data includes a first XML documentidentifier for the first XML document; wherein the first data isuniquely identified by a first Unique Annotation Link Identifier (UALI);wherein the document system is configured to respond to a requestcontaining the first XML document identifier with a manifestation of asecond instance of the first XML document; the method comprising: (A)receiving a first request containing the first UALI; (B) issuing asecond request to the document system, wherein the second requestcontains the first XML document identifier; and (C) in response to thefirst request containing the first UALI, manifesting the first data inconnection with the manifestation of a second instance of the XMLdocument.
 10. The computer system of claim 9: wherein the first datacomprises first annotation data representing a first associationbetween: (a) a first manifestation of first annotation content inconnection with a manifestation of the first instance of the first XMLdocument and (b) a manifestation of a first instance of a first targetin the manifestation of the first instance of a first XML document;wherein the first UALI comprises a first Unique Annotation Identifier(UAI) that uniquely identifies the first annotation data; wherein (A)comprises receiving a request containing the first UAI; and wherein (C)comprises, in response to the request containing the first UAI,manifesting the first annotation data in connection with a manifestationof a second instance of the first XML document.
 11. The computer systemof claim 10, wherein the first annotation data comprises firstannotation content data representing the first annotation content, firsttarget locating data for locating the second instance of the firsttarget, and first association data representing the first association.12. The computer system of claim 9, wherein the first data comprisesfirst annotation set data representing a first annotation set; whereinthe first UALI comprises a first Unique Set Identifier (USI) thatuniquely identifies the first annotation set; wherein (A) comprisesreceiving a request containing the first USI; and wherein (C) comprises,in response to the request containing the first USI, manifesting thefirst annotation set data in connection with a manifestation of a secondinstance of the first XML document.
 13. The computer system of claim 12,wherein the first annotation set further contains second annotation datafor annotating the first instance of the first XML document; and wherein(C) further comprises, in response to the request containing the firstUSI, manifesting the second annotation data in connection with themanifestation of the second instance of the first XML document.
 14. Thecomputer system of claim 9, wherein (A) comprises: (A)(1) at a clientdevice: (A)(1)(a) receiving input containing the first UALI; (A)(1)(b)retrieving the first data from storage; (A)(1)(c) requesting the secondinstance of the first XML document from the document server; and(A)(1)(d) manifesting the first data in connection with themanifestation of the second instance of the first XML document.
 15. Thecomputer system of claim 9, wherein (A) comprises: (A)(1) at a clientdevice: (A)(1)(a) receiving input containing the first UALI; (A)(1)(b)transmitting a request containing the first UALI to an annotationserver; (A)(2) at the annotation server: (A)(2)(a) requesting the secondinstance of the first XML document from the document server; (A)(2)(b)creating a composite of the first data and the second instance of thefirst XML document; (A)(2)(c) transmitting the composite to the clientdevice; and (A)(3) at the client device: (A)(3)(a) manifesting thecomposite.
 16. The computer system of claim 9, wherein (A) comprises:(A)(1) at a client device: (A)(1)(a) receiving input containing thefirst UALI; (A)(1)(b) transmitting a request containing the first UALIto an annotation server; (A)(1)(c) requesting the second instance of thefirst XML document from the document server; (A)(1)(d) transmitting thefirst data and the second instance of the first XML document to theannotation server. (A)(2) at the annotation server: (A)(2)(a) creating acomposite of the first data and the first XML document; (A)(2)(b)transmitting the composite to the client device; and (A)(3) at theclient device: (A)(3)(a) manifesting the composite.